Semantic search is one of the most practical applications of generative AI in real data processing projects. In this course, we go beyond the basic introduction to embeddings (from the course The Hidden Foundation of GenAI) and start using them in practice. You will learn to build a complete semantic search pipeline from scratch: from creating embeddings and storing them in a vector database to performing natural language queries.
The course is built around a real data observability project. You will create a pipeline that collects logs, processes them using FastAPI, and stores the embeddings in qdrant - a high-performance vector storage. Then, you will develop a dashboard on Streamlit, allowing you to search logs by meaning, rather than by keywords, and compare the results with traditional SQL queries in DuckDB.
Key steps of the course:
- From embeddings to search: review the basics of embeddings and analyze how exactly they enable semantic search functionality.
- Building a pipeline: implementing an API on FastAPI for log processing and embedding generation.
- Working with qdrant: collections, points, cosine similarity search, and optimization of embedding structure.
- Streamlit interface: creating a user-friendly search and comparing the semantic approach with classic SQL.
- Improving accuracy: methods for optimizing embeddings, query formulation, and search configuration.
- Launching in Docker: deploying the entire stack (FastAPI, qdrant, Streamlit, DuckDB) using Docker Compose.
- Bonus: using DuckDB for analytics - implementing WAL, working with data in Docker, and comparing the capabilities of SQL and vector search.
Upon completion of the course, you will not only understand the mechanics of semantic search but also have a ready-to-use working project that can be adapted for your own AI-based solutions.
