Idea: Crawlee-friendly “docs → clean Markdown for RAG” pipeline (example + config)

[HEDELKA]

Hi Crawlee community!

I’m building a docs-focused scraper that outputs clean Markdown for RAG pipelines (navigation stripped, code blocks preserved), with metadata + optional chunking.

Proposal: publish a Crawlee-friendly example pipeline:

• crawl docs pages (Crawlee)
• extract/normalize main content into clean Markdown (RAG-ready)
• load into LangChain/LlamaIndex/vector DB

Questions:

1. Would an example like this be valuable for Crawlee users building AI assistants / knowledge bases?
2. Where should it live (Discussions, examples repo, blog)?
3. Any preferred conventions for metadata fields (source URL, section path, etc.)?

I can share a minimal working config and a small sample output to validate the approach.

[janbuchar]

Hi @HEDELKA and thank you for using Crawlee! We'll be happy to accept a PR with such an example. The location depends on the format of the article, but both a blog post (https://github.com/apify/crawlee/tree/master/website/blog/2025) or a documentation guide (https://github.com/apify/crawlee/tree/master/docs/guides) are viable options.

If you're going to go end-to-end from crawling to integration with LangChain/..., perhaps a blog post would be the best fit. Would you mind joining us at Discord and going over the details with @souravjain540?

[souravjain540]

@HEDELKA i am waiting for your reply on discord :) cheers!

[HEDELKA]

Hi @souravjain540!

So sorry for the silence. Ironically, my Discord account got auto-banned by their spam filters right after I joined the server (likely a false positive). I currently don't have access to chat there.

Could we coordinate via email instead? You can reach me at: HEDELKA@yandex.ru

I have the draft structure ready for the blog post.