Yesterday, in San Francisco, OpenAI held a DevDay to introduce its new generative AI technologies to developers. And this week, Google Cloud released a trove of new features and enhancements to its line of cloud storage offerings to help devs get to the data faster for their AI projects.
There are 10 times more software developers in the world than data scientists, noted Andi Gutmans, Google’s general manager and vice president of engineering for databases, citing the Bureau of Labor Statistics in a blog item listing the new technologies that were posted Wednesday.
The company has brought vector processing support to its PostgreSQL and Redis/Valkey-based managed services, and a fully relational query capability to Firebase. Spanner has been updated with modern AI technologies as well.
Vector Processing for PostgreSQL
There are few databases more popular with developers than PostgreSQL these days, and Google has taken note of this, offering its own AlloyDB, a database service with 100% PostgreSQL compatibility.
This week, the company has infused AlloyDB with its ScaNN vector index, which has long given Google Search and YouTube the ability to search large realms of unstructured data, such as text and video, through vector embeddings.
Vector processing is the magic key to bringing standard relational databases into the world of generative AI.
In a relational database, vector embeddings can represent unstructured data such as text or video metadata. They can be stored in a database and located by way of a vector proximity operation, a “nearest neighbor” query generated from a user request.
“The ScaNN index is the first PostgreSQL-compatible index that can scale to support more than one billion vectors while maintaining state-of-the-art query performance — enabling high-performance workloads for every enterprise,” Gutmans wrote.
Google is also expanding AlloyDB across competitors’ clouds. The company has partnered with AI platform provider Aiven, which will allow customers to expand their AlloyDB database across different clouds and in their own facilities through the Aiven for AlloyDB Omni, a managed cloud database service.
This service is built from Google’s AlloyDB Omni, a downloadable edition of AlloyDB.
“You can now run transactional, analytical, and vector workloads across clouds on a single platform and easily get started building gen AI applications, also on any cloud,” Gutmans wrote.
Vector Processing for Redis and Valkey
Vector processing is also coming to the Redis low-latency, low-latency database, often used for caching data that must be delivered quickly, as well as to the Linux Foundation-backed open source fork of Redis called Valkey.
Google Cloud has a managed service for both Redis — called Memorystore for Redis Cluster — and for Valkey, Memorystore for Valkey 7.2.
With this new technology, a single Memorystore instance “can perform vector search at single-digit millisecond latency on over a billion vectors with greater than 99% recall,” boasted Google engineers in a recent blog post.
Users could potentially scale out their database across 250 shards, allowing them to store and search billions of vectors.
‘The key to this performance and scalability is partitioning the vector index across the nodes in the cluster,” wrote Google Cloud product manager Kyle Meggs and Google software engineer Jacob Murphy in the post.
“Memorystore uses a local index partitioning strategy, meaning that each node contains a partition of the index that corresponds to the portion of the keyspace that is stored locally,” they wrote.
The keyspace is already sharded uniformly thanks to the Redis cluster protocol, so each partition is about the same size.
“Because of this design, adding nodes linearly improves index build times for all vector indices.”
The company has also launched a public preview of Memorystore for Valkey 8.0, based on the first major release of Valkey that differs from Redis in a number of significant ways, including a new replication scheme and more visibility into performance and resource usage.
PostgreSQL for Firebase
Developers using Google’s own Firebase platform for their data storage needs get some PostgreSQL love as well.
This easy-to-use data store now gets relational capability like standard databases such as PostgreSQL, with the new Firebase Data Connect, now in preview.
In fact, Data Connect is actually a fully managed PostgreSQL database powered by Cloud SQL. For each new database needed, Data Connect automatically creates a database schema, secure API server, and typesafe-generated SDKs for Android, iOS, Web, and Flutter applications, according to Google.
Also on the Docket
The company’s Spanner big data database has also been updated for AI work. It has been integrated with the LangChain model. Spanner Graph has been added for interconnected data, advanced full-text search, and vector search.
Spanner is now included in Google’s Database Center, a service that offers a single pane of glass for monitoring and operating database fleets at scale. Cloud SQL and AlloyDB will soon be supported. Database Center also now features AI-driven intelligent chat, cost optimization and advanced security recommendations.
Learn more about Google Cloud databases here and about Google’s Data Cloud here.
The post Google Equips PostgreSQL, Valkey Services for Vector Processing appeared first on The New Stack.
The company has brought vector processing support to its PostgreSQL and Redis/ValKey-based managed services, and a fully relational query capability to Firebase. Spanner has been updated with modern AI technologies as well.