Slashot: Converting Oracle to Postgres

Have you ever worked for a truly large company? I ask because you seem to trivialize the politics of the environment.

You are talking about cushy jobs for most of these people so there is incentive to CYA. You also have separate teams who report to separate managers who each control a layer in the application. You have the dba teams, the mainframe teams, the noc team, the platform team, the framework team, the other framework team, the application teams, the qa teams, the internal client teams, etc. If you looked at it from their perspective these people don’t necessarily want to allow some wet behind the ears application team (because thats usually who are working at this layer anyway or worse offshore) to have so much control over what is essentially very proprietary business information. Can you really blame them though? If you’re some medical company who deals with patient information, and you have HIPAA obligations, perhaps it can start to make sense? Even worse if you are publicly traded because then you have to deal with SOX.

Not to mention that there are many positive reasons to use stored procedures in general. Such as the ability to encapsulate your data structures in the database allowing you to change schema without affecting the application layer. Or allowing DBAs to identify areas to increase performance through indexes, etc. since they know every single query being run on their database. Or simply reducing round trips between the application layer and the database layer. Or increasing quality of code by inherently using transactions thereby hopefully reducing times when the database is in an incorrect state and not relying on an application developer to get that right. Also creates a uniform platform when you have multiple application teams. What about simply using stored procedures allowing your application to potentially switch database software with minimal code change, if written correctly.

There are many good reasons to use stored procedures.

.. You’re omitting a key concept about price that I’ve seen many fools around me exploit. If they are managing a big project that has software costs of millions a year then they must be very important. If I’m managing a project that only costs the company a hundred thousand or so in software costs a year then I must be less important. It really doesn’t seem to matter if my project does more or is actually more critical to the operation of the company, those are concepts that are too abstract for upper management to understand. So if you want your department to look important then waste as much money as you can.

 

Pipeline DB: The Continuous View

Although in recent years the general conception of data has begun a phase shift: it was once a solid to be stored and pulled out of a silo on demand and it is now becoming a fluid that continuously surges through complex networks of cooperating systems, enabling them or their users to do smart things as soon as information is produced.

.. This phase shift has led to the beginning of the displacement of classical data technologies by next-generation systems that are able to refine and understand information in real time.

.. We think that the primary consumer of information is becoming software itself and not necessarily analysts who need to frequently run dynamic, ad-hoc queries on granular data. And applications — almost by definition — are pre-programmed to issue certain types of queries to the database, with the only dynamic aspect being parameters such as date ranges.

The Streaming SQL Database

An open-source relational database that runs SQL queries continuously on streams, incrementally storing results in tables.

No ETL

PipelineDB eliminates the ETL layer. Stream granular data directly into the database and continuously distill it with the SQL queries you’ve declared.

Continuous views are the core abstraction of PipelineDB. You can think about them as very high-throughput, realtime, incrementally updated materialized views. The most important property of a continuous view is that only its output is actually stored in the database.