John Wilkin has another nice post, talking about making resources available in such a way as to make them more likely to be crawled by Google and hence more generally discoverable and, importantly, useful and used.
We often go wrong, however, when we try to share our love of complexity with the consumers. We've come to understand that success in building our systems involves making complicated uses possible without at the same time requiring the user to have a complicated understanding of the resource. What we must also learn is that a simplified rendering of the content, so that it can be easily found by the search engines, is not an unfortunate compromise, but rather a necessary part of our work. [John Wilkin’s blog » Our hidden digital libraries]
Roy has been talking about this issue also.
This is clearly less straightforward than many imagine. Google can make choices about what to crawl, what to index, and what to present in results. At play for larger sites also is the danger of falling foul of the search engines' spam protection measures.
I participate in a JISC advisory committee on repository issues in the UK. I spent some time arguing earlier this year that search engine optimization should be a higher priority for repository managers, for institutions, and for both the capacity building systemwide infrastructure and advisory structures that JISC is capable of providing. We now recognize that simply having stuff on the web is often not enough. It is increasingly necessary to think about how well it is being crawled, indexed and discovered. .
This is why I have emphasized 'disclosure' as a new word in our service lexicon. We may not control the discovery process in many cases, so we should be increasingly concerned about effective disclosure to those discovery services. Effective disclosure has be be managed, whether it is about APIs, RSS feeds, support for inbound linking, exposure to search engines, ......