Method5

Avoiding Oracle Quines

2018-06-09T00:00:00+00:00

Unexpected quines can appear when querying tables like V$SQL. To avoid them you must sometimes add an extra predicate to those queries:

	select *
	from v$sql
	where ...
		and sql_fulltext not like '%quine%';

What is a Quine?

A quine is a program that outputs its own source code. Normally it’s a form of code golf; a challenging, impractical puzzle to entertain programmers. They’re usually hard to create and look cryptic.

Here’s a Python 2 example.

	_='_=%r;print _%%_';print _%_

See this Wikipedia article or this Code Golf question for more details and examples of quines.

How Do They Happen?

Quines can happen when querying V$SQL, because the query will match its own conditions. For example, this query searches for all recent SQL statements that reference the table “FOO”:

	SQL> select * from foo;

	no rows selected

	SQL> select sql_id, sql_text from v$sql where lower(sql_fulltext) like '%foo%';

	SQL_ID        SQL_TEXT
	------------- --------------------------------------------------------------------------------
	83zk38bvaq4nu select * from foo
	2zadv09qpyc9a select sql_id, sql_text from v$sql where lower(sql_fulltext) like '%foo%'

	2 rows selected.

Before the query against V$SQL was run, it was parsed and put in V$SQL, which is why it found itself. These recursive problems are weird and take a while to get used to.

How Do We Stop Them?

Stopping the quines is easy. Simply add another predicate that will exclude itself:

	SQL> alter system flush shared_pool;

	System altered.

	SQL> select * from foo;

	no rows selected

	SQL> select sql_id, sql_text from v$sql where lower(sql_fulltext) like '%foo%' and sql_fulltext not like '%quine%';

	SQL_ID        SQL_TEXT
	------------- --------------------------------------------------------------------------------
	83zk38bvaq4nu select * from foo

	1 row selected.

That extra predicate will look weird at first, but you’ll get used to it.

Why Does it Matter?

There are many times when it’s necessary to search for SQL statements. Usually it’s for ad hoc tuning or debugging, when it doesn’t make a big difference if there is one extra row.

But occasionally you need to find one, and only one SQL statement, so you can do something with it.

For example, sometimes in Oracle you want to force a hard-parse. Normally it’s great that Oracle remembers your query, and can learn more on each execution. But sometimes you don’t want it to remember anything.

This can happen when you are running dynamic SQL in SQL, and the return types are not known until run time. Since the return type can change over time, even for the same input, it’s important to flush the statements from the shared pool. And to save time it helps to only flush the one statement that matters.

This is used in Method5 because the function M5 need to be re-parsed every time, in case it will return different values. You can see this in action in the procedure method5.m5_purge_sql_from_shared_pool. (There are a bunch of small objects in that file, search for “quine” to find it.)

It’s not a common problem, and not incredibly important. But it’s interesting to see spontaneously happen when it’s usually quite difficult to create intentionally.

Improve Entity-Attribute-Value models with proper data types

2017-11-14T00:00:00+00:00

Entity-Attribute-Value (EAV) models are a contentious topic in database design. Flexibility is important but some things should not be sacrificed.

EAVs should use a table like this:

	NAME          VARCHAR2(4000)
	NUMBER_VALUE  NUMBER
	STRING_VALUE  VARCHAR2(4000)
	DATE_VALUE    DATE

EAVs should avoid a simpler table structure like this:

	NAME   VARCHAR2(4000)
	VALUE  VARCHAR2(4000)

Those extra columns don’t take away any of the benefits and alleviate many of the costs.

Pros and Cons

Here are some common arguments in EAV debates:

Pros

Flexibility to add anything without schema changes.
Looks nicer on the schema diagrams.

Cons

Data type errors
Bad performance.
Lack of constraints.

Architects and application developers prefer EAV and DBAs tend to hate it.

The best design is a compromise. Be honest about the importance of each pro and con. While it is important to compromise on high level schema designs we should never compromise on low level type safety.

Flexible

Applications can still store anything. Those 3 columns cover 99% of all data. It’s easy to add more columns if necessary.

Some people might complain that those extra columns create more work because you need to identify the type. But in practice, if you have a single value you will almost always need to know the type anyway. Even if the value will only be displayed on a screen the type is useful for justifying the text.

If you truly have unstructured data it should still have a data type, such as JSON or XMLType.

Looks Nice

This reason usually isn’t that important. Developers and DBAs typically use an IDE as the primary interface for the database, not the PDFs generated by the data modeling tool.

But with the compromise solution this isn’t an issue, it only requires a few extra columns one time.

Data Type Errors

This is a subtly dangerous issue that you may not discover until it’s too late.

It is difficult to control the order a SQL query is executed. Without that guaranteed order most developers will write unsafe queries like this:

	select *
	from eav
	where name = 'Date of Birth'
	and value = date '2000-01-01';

That query will usually work. But it is not guaranteed to work because Oracle may check the value predicate before the name predicate, and not all values can convert to dates.

There is only one safe way to write that query:

	select *
	from
	(
		select *
		from eav
		where name = 'Date of Birth'
		and rownum >= 1
	)
	where value = date '2000-01-01';

That is the only safe way because ROWNUM is a special function that Oracle assumes means “these rows must be displayed in order”. Oracle will never push a predicate or view merge into an inline view with a ROWNUM.

No other fix will work reliably. A simple inline view can be trivially rewritten by the optimizer. The physical order of predicates is ignored. Hints are difficult to get right. Even a CASE statement does not always guarantee short circuit evaluation in SQL (despite what the manual says).

Use the extra type-safe columns unless you want to write the ROWNUM trick a hundred times and are willing to thoroughly warn all SQL developers about the dangers in your data model.

Bad Performance

The extra type columns can significantly boost performance.

The exact type will be smaller and slightly faster to compare than a converted string.

The optimizer will have a little better information with the right types.

Most importantly, the right types let you avoid the horrible ROWNUM trick. That gives the optimizer more flexibility to transform the query and choose a better plan.

Lack of Constraints

Using the right types doesn’t allow for smart constraints but at least it supplies some basic type checking.

Validating dates and numbers is much harder than most developers realize.

For dates, very few applications properly deal with internationalization. For example, the month abbreviations are not the same in every language.

For numbers, very few applications can handle all number formats. Number validation sounds simple until you look at the Oracle manual’s complex syntax diagrams. (Tip - those diagrams are helpful but contain a mistake. If you can’t find it, don’t try to write a regular expression to handle number validation). The Method5 PL/SQL lexer uses this regular expression. But don’t use it, this is only an example of how horrible the regular expression will be. It also doesn’t work with “+” or “-“, because the lexer considers those operators and not part of the number.

	([0-9]+\.[0-9]+|\.[0-9]+|[0-9]+)((e|E)(\+|-)?[0-9]+)?(f|F|d|D)?

Always Use the Right Type

Using the default Oracle data types costs almost nothing, avoids type errors, improves performance, and simplifies validation. Adding a few extra columns now can prevent a lot of problems later.

The Difference Between Debugging and Performance Tuning

2017-10-18T00:00:00+00:00

On the surface both performance tuning and debugging follow the same process: find a problem, drill down to the root cause, then fix or work around the underlying problem.

But there’s a key difference between the two. Programmers and administrators who don’t understand the difference tend to waste a lot of everyone’s time when they try to performance tune.

Debugging

When debugging you only need to find the first problem and then drill down. The program must work 100% correctly, if even a single byte looks wrong then it should be investigated. Even if you don’t fix the problem you set out to fix you still accomplished something useful.

Tuning

For performance tuning you need to find the first significant problem, and then drill down. It’s easy to get distracted by weird behavior and wait events. If you don’t fix the right problem you’ll waste a lot of time.

The art of performance tuning is to gather relevant data and then ignore the 90% of the process that only accounts for 10% of the runtime. And then ignore the 9% that is slow but is necessarily slow.

Motivated Troubleshooting

There are many performance tuning dead ends. I’ve seen many people, including myself, waste days fixing a problem that could not theoretically improve performance by a significant amount.

We all have the urge to take the easy path and fix problems we are familiar with. For bug fixing that’s usually a good idea - any cleanup is a good thing. It can be helpful to have a utopian view of the code.

But performance tuning must stay firmly grounded in reality. And the ugly truth is that nobody will ever fully understand all the wait events or all the execution plan operations. We only have time to fix the slow ones. Don’t bother tracing the entire process, sampling is good enough.

For personal education it’s helpful to investigate anything that doesn’t look right. But when there’s a critical performance problem you must focus on the most important parts.

Use SQL Styles to Reduce Complexity

2017-06-30T00:00:00+00:00

Use these slightly unusual styles to reduce the complexity of SQL code:

Avoid Abbreviations
Avoid Aliases
Use Inline Views
Tab Indenting
Lower Case

These rules make sense when SQL code complexity is measured by something other than a simple character count, and when the legacy ideas of SQL are abandoned.

Avoid aliases and abbreviations - words are the atomic unit of code

A word count is a better measure of complexity than a character count.

Drilling down through the hierarchy of a program can go through many different paths. A PL/SQL program has objects, objects have statements, statements have SQL, etc. It’s easy to end up at a low-level concept like a character, byte, bit, etc. But those concepts are too low level.

Only complexity as perceived by a human is important. In general, people think and operate on words, not characters. We read words - except for weird words or children learning to read. We type words without even thinking about the letters - except for people who don’t know how to type. A large word is simpler than a few random letters put together.

This statement:

	select *
	from employee
	where employee.name = 'John Smith';

Is simpler than this statement:

	select *
	from employee e
	where e.name = 'John Smith';

The first version has more characters but fewer words. It also has one less variable, the “e”. This is why aliases should typically be avoided.

Now compare this statement:

	select * from employee;

With this statement:

	select * from emp;

Again the first statement has more characters. But all English speakers instantly know what an “employee” is. Reading the full word requires virtually no attention. The abbreviation “emp” is simple but it still requires a tiny amount of thought. Why make people think more than they have to?

Abbreviations save characters but require more mental effort and take longer to type. Unusual abbreviations should be avoided.

Obviously the definition of “unusual” is subjective and depends on the context. If every programmer on your project instantly recognizes an abbreviation then use it. But don’t invent new ones for every SQL statement and table and don’t follow some “abbreviation guideline” that generates cryptic code.

Use inline views - measure joins with chunks and factorials

Once a simple character count is abandoned it is possible to think about code in more logical ways.

Compare the one-join-clause-to-rule-them-all approach:

	select * from t1, t2, t3, t4, t5, t6 ...

With an inline view approach:

	select * from
	(
		select * from t1, t2, t3 ...
	) view1
	join
	(
		select * from t4, t5, t6 ...
	) view2
		on view1...

(Ignore the old-fashioned Oracle join syntax for now. ANSI joins are superior but that’s a separate topic.)

The second version is not only bigger but has additional words, additional joins, and even repeats things. Yet it is simpler for several reasons.

First, think of inline views as the equivalent of procedures in a procedural language, or functions in a functional language. Sets are the building blocks of SQL. There is an elegance to nesting them and creating hierarchies of sets. You wouldn’t use a single, ginormous procedure to do everything in a procedural language, don’t make a similar mistake in SQL.

Next think about the different ways to join the tables. There are 6! (6 factorial, or 654321) ways to order the 6 table join, so there are roughly 720 ways to mentally visualize the tables. Compared to 3!2, or roughly 12 ways to visualize the inline views.

I wouldn’t say that version 1 is literally 720/12 times more complex than version 2. But the complexity of adding tables to a join clause is not linear. We all intuitively know this - at some point a function just gets too big. It’s the same with join clauses.

It can be helpful to think of code in terms of chunking. Chunking is the idea that our minds constantly group things into chunks and we can only keep a small number of chunks in short term memory at the same time. Research and common sense suggests the number is small, like between 5 and 7.

So build small, intuitive chunks of SQL using inline views. Join those chunks and repeat. This only works with inline views because inline views take no input and have no side effects. Think of them as pure functions.

This benefit does not extend to correlated subqueries and the WITH clause. Those features often make code more complex because they introduce side effects and tie everything together. Both obviously have their place in SQL code. But if you always use WITH clauses just to make your code read top-to-bottom you’re adding a lot of complexity.

Tabs instead of spacing - indenting is for showing parent-child relationships, not random text alignment

	select ...
	from ...
	where ...

vs.

	select ...
	  from ...
	 where ...

To some people the spaces versus tabs debate boils down to: tabs are sometimes faster but spaces can make things look prettier.

In general I believe in spending the extra time to make code look good but not in this case.

First, the speed different between tabs and spaces is greater in SQL. This is because SQL auto formatters suck and should be avoided (because SQL is too complex to parse, but that’s another topic.) And beautifiers and IDEs won’t help you format dynamic code that looks like a string. And in PL/SQL dynamic code is critical. Avoiding dynamic SQL leads to some horrible anti-patterns.

The beauty of spaces is an illusion. The point of indenting is to show parent-child relationships and make it clear what code is on the same level. Indenting with tabs and left-justifying keywords makes it abundantly clear what those relationships are.

On a slightly related topic, this is why inline view parentheses should be on their own line like this:

	select *
	from
	(
		select ...
		from ...
	)

Those parentheses are so important they deserve a separate line. The code between them is an entire state and can exist separate from everything else in the query. That inline view can be copied and pasted or debugged all by itself. Inline views are the key to SQL - give them some space.

Use lower case - because SQL is important

Upper-casing SQL probably started because that’s how COBOL was written. Today’s SQL is much better than it was in the 1980s, it’s time to break with the past and stop typing like an angry teenager.

One current advantage to using upper case for SQL is to help contract it with other languages or writing. For example, when quoted in this paragraph it looks better to use SELECT * FROM DUAL; There’s no mistaking that SQL statement for a regular sentence. The same applies when embedding SQL as a string in languages like Java and C#.

It makes sense to use upper case for small embedded strings. But SQL doesn’t need to be relegated to that role. If you’re a database programmer then SQL should be your main language. Your SQL should look good in isolation. No other modern language uses upper case, using upper case for PL/SQL programs makes them look old fashioned.

Style matters

Code that looks good will be respected more, maintained better, and is more likely to be reused. I think most good programmers appreciate beautiful code and enjoy working on things that look good.

There’s more than one way to do it and no one should look down on code just because it uses upper case, or spaces instead of tabs. The important thing is that database programmesr ask themselves, “why am I writing code like this?” Most SQL code is optimized to save bytes and adhere to tradition.

Code should be optimized for the most important resource - our attention. Spend extra time to write code that is easier to read.

Simplify string concatenation with alternative quotes and templates

2017-05-12T00:00:00+00:00

Combine the alternative quoting mechanism and templates to dramatically improve code readability compared to simple string concatenation.

Example

Here’s a simple example that uses the alternative quoting mechanism to avoid escaping single quotes:

	execute immediate1 q'[ create database link ... using 'some_tns' ]';

The alternative quoting mechanism allows almost any character to become the quote. And the characters {,[,(,< are matched to },],),>. This makes it much easier to embed code inside code - there’s no need to change all single quotation mark.

Here’s a simple example of using a template for putting variables into a string:

	execute immediate replace(q'[
		create database link using '#TNS#'
	]', '#TNS#', v_tns);

There are template engines but a simple replace function works well enough. This style allows keeping the dynamic code simple and as true to the real value as possible. Dynamic code is complicated, adding a bunch of concatenation makes it even more complicated.

Here’s the concatenation version:

	execute immediate 'create database link ... using '''||v_tns||'''';

The concatenation version has less characters but the single quotes are ugly and confusing. And this is the simplest possible example, it only gets worse.

For a more realistic example, take a look at this SQL template in Method5. It’s code that runs code, that runs code, that runs code. It’s rare to nest dynamic code this many levels deep but sometimes it’s necessary. The code is confusing enough with templates and alternative quotes. Imagine what it would look like with hundreds of extra single quotes.

The alternative quoting mechanism and templating makes it possible to embed code inside code, even multiple levels deep, and keep the code sane.

Why Not Use Other Features?

You may be thinking, “don’t use dynamic code”. Features like reflection and polymorphism work great in most languages and can eliminate the need for dynamic code.

Unfortunately those features don’t work well in PL/SQL. The top Google result for “PL/SQL Reflection” is one of my Stackoverflow answers. I’m not bragging - that doesn’t prove I’m an expert in PL/SQL reflection, it proves that there are no experts in PL/SQL reflection. (Don’t bother Googling that answer - the result is ugly and almost certainly not helpful.)

Luckily Oracle does a good job with dynamic SQL and PL/SQL. It’s fast and convenient and improves with every release. You don’t want to build overly generic code in PL/SQL. It’s better to write code that builds specific code.

Make the Code Readable

It’s easy to over-look the human side of code and ignore readability. The program only works if the compiler is happy. And the compiler doesn’t care if it looks bad.

But the real user of all code is other programmers, not the compiler. Dynamic SQL and PL/SQL are powerful but they are also confusing and require extra care. Use templates, the alternative quoting mechanism, comments, and pay extra attention to formatting to make that code as beautiful as possible.

If it looks right it will run right.

Avoid Java in Oracle Database

2017-04-12T00:00:00+00:00

The primary languages of Oracle databases are SQL and PL/SQL. Despite what many official sources claim, Java in the database is a distant third and should be avoided when possible.

This is not a Java rant and I am not trying to say anything negative about Java as a programming language. Obviously it’s a popular and good choice for working with databases. Regardless of how good and popular Java is, there are several reasons to avoid using Java stored procedures inside an Oracle database.

Java is Not Always Available

The main problem with using Java inside an Oracle database is that Java is not always available. Oracle owns Java and has been advertising Java-in-the-database for decades, but in practice you cannot rely on it being installed.

Java is not available for the Express Edition. I’m not a big fan of Express Edition, but a lot of people use it, so this is a show-stopper for many products.

Java is an optional component in all other editions. It may be a default, but a lot of DBAs like to remove it because it makes upgrades slower. Installing components is not a trivial task. It often requires a DBA and downtime, and in many organizations that would require a lot of paperwork and justification.

Java Does Not Fit Perfectly

It’s unfair to fault a product for not integrating perfectly. But when SQL and PL/SQL integrate so seamlessly, it’s hard not to look at Java stored procedures with disgust.

Of course the types don’t completely match. There’s probably all kinds of fun issues with string sizes, character sets, numeric precision, etc.

Java objects can be larger than 30 characters longs, which causes some interesting naming problems. (Although that problem may be fixed in 12.2 with larger names.). The objects look different in the data dictionary and virtually all DBA scripts will fail to properly handle Java objects. This can be fixed by translating the names with DBMS_JAVA.LONGNAME but it’s a pain.

There’s a culture clash between Java and database programmers. Database programmers want code to “live” in the database, everyone else in the world wants code to “live” in version-controlled text files. 99% of all database programmers are clearly, horribly wrong here. But you can’t convince them (believe me, I’ve tried), and they “own” the database.

Some of these issues may not apply to SQLJ. SQLJ seems like a neat way to integrate Java and SQL to me, but I’m in a small minority there. I get the feelings that embedding SQL died in the 90s and nobody wants that kind of product anymore.

SQL and PL/SQL are Almost Always Better Choices

If you’re doing something inside a database then SQL is almost always the best language. Most people are only familiar with the basics, for running select * from employee;. But Oracle’s SQL syntax is incredibly rich, there’s hardly any programming task that can’t be handled well with SQL.

I’ve been hired a few times as a “PL/SQL Developer”. What that title really means is that I wrote SQL and tied it together with PL/SQL.

Yes, Java has more powerful language features than PL/SQL. But they are completely irrelevant 99% of the time. A good Oracle database program will do almost all of the heavy lifting in SQL. The procedure language just needs to tie it together. So PL/SQL’s improved integration features more than make up for being a less powerful language.

This is a Shame

Oracle is missing a huge opportunity here. It would be so easy to fix - just make Java a required component in all editions!

Until that gets fixed that means you almost certainly should not develop a product with Java inside the database. This is a shame because there are at least a few cases where it would be incredibly useful to call Java from the database.

I would love to be able to reliably take advantage of Java’s advantages:

A more powerful development kit that helps for many tasks, like creating GUIDs, file processing, etc.
Run existing Java programs, like ANTLR.

For the foreseeable future it’s best to do all database processing in SQL and PL/SQL, even if those languages don’t seem like a perfect fit for the task.