Tuesday, June 27, 2017

Primary Key columns in INCLUDE list

I was reviewing someone's code the other day and I saw they had put primary key columns in the INCLUDE list of a non-clustered index

-- Create a table
if object_id('tempdb.dbo.#PhoneBook') is not null drop table #PhoneBook
create table #PhoneBook
    FirstName nvarchar(500),
    LastName nvarchar(500),
    PhoneNumber varchar(20),
    DOB datetime

create clustered index #IXC__#PhoneBook__LastName on #PhoneBook (LastName)

create nonclustered index #IXN__#PhoneBook__PhoneNumber on #PhoneBook (PhoneNumber)

A little background on the physical structure of non-clustered indexes for those who might not know. A non-clustered index consists of at least two, and up to three components. All non-clustered indexes store the index key (in my example above, this would be [PhoneNumber] in the nonclustered index) as well as a pointer back to the clustered index (in this case, the columns [LastName]).

The Phone Book example is the classic indexing example thrown around. The clustered index is the way the data is physically structure. In a phone book, that's usually by last name. But if you wanted to look something up by a different field, say PhoneNumber, you probably wouldn't want to re-print an entire phone book ordered by PhoneNumber. Instead, you could create an index in the back of the book which lists phone numbers, and a reference to the page you could find those phone numbers in the main part of the book.

So if you wanted to look up the first and last name of the person with the phone number 555-555-5555, it might say "the last name of the person with this phone number is 'Smith'. You would then flip to the page that has the "S"s and start looking for Smith. (note, this is called a "clustered index lookup" when performed in SQL), and then read the persons first and last name.

At a minimum, that index has to have the index key (PhoneNumber) and where you can find that phone number in the phone book. But now say I don't want to have to flip back through the phone book to see the person's first name. I could annotate the index to also contain the first name of the person associated with that phone number so that I never have to flip back through the book looking through all the "S" names. That's what INCLUDE column's are. They're not information you can search on, they're just bits of data from the clustered index which are copied to the index structure so you don't need to do that lookup on the primary key.

So now we get back to the matter at hand. What happens when you include a column that's part of the primary key? Short answer, nothing. Those column's are already part of the index's physical structure because of the pointer from the index to the clustered index. SQL is smart enough to realize this, and doesn't copy those values twice or have to do extra maintenance on them. So the size and overhead of the index is the same.

So should you include them or not? You may also have noticed that if you look at an execution plan, SQL will often suggest indexes which include clustered index. While I'm not personally a fan of including those columns, there isn't any performance hit for doing so. And it was pointed out to me that if you DO explicitly include columns from the primary key to cover a query, should that primary key change, those columns will STILL be covered. There's something to be said for that, but if you're changing a primary key on a production table, that may be the least of your worries.

So what did we learn? INCLUDE-ing primary key columns in a non-clustered index is totally unnecessary as those references already exist. However it also does no harm to the performance of queries or the overhead of maintaining that index. It may just come down to whether you'd like to have your index INCLUDE list nice and tidy, or if you think there's value to explicitly stating that those columns need to be covered. Personally, I prefer tidy, but I can see it the other way too.

Hopefully you now know a little more about what the end result of doing this is.

Tuesday, March 14, 2017

Rolling an Idea Around in Your Hand

I hear from many people (my wife included) that they “just don’t have a mind for programming”. Now maybe that’s just a polite excuse to say they have no interest in programming, which, too, is fine. And while, like with music, there are some people who really have (or do not have) structures in their brains making it difficult if not impossible to do, I think the majority of the skills are generic enough to be learned with practice and more importantly, a drive to do so.

One of the skills you pick up along the way is what, by analogy, I’d like to think of as rolling an object around in your hand. There are few things in our daily lives that we’ve never seen before, let alone something you can hold in your hand. But there are some. Imagine an unsolved Mirror Cube. Or maybe you’ve decided you want to work on your car, and you start removing parts you never knew existed. Or maybe you get a Christmas present still in the packaging and you’re trying to figure out what it might be?

Many times, you’ll just start fiddling with it. Feeling its surface. Rolling it around in your hand. Feeling its weight. Inspecting it from different angles; color, shape, function. If it has parts, how they fit together. Maybe the parts move. Maybe they move and they shouldn’t. Maybe, and most cripplingly, you’re afraid to do the above because you think you might break it, or not be able to put it back the way it was. Maybe you’re right.

The point is, you’re solving a problem. You don’t realize it yet, but you are. Any time you’re presented with something you don’t fully understand you start doing an investigation. The only difference in the software world is you’re lucky enough to be able to hit the undo button, removing much of the risk in the last bit of the previous paragraph.

To a large extent, that’s what software development is. Or more accurately, that’s what software maintenance and troubleshooting is… which is basically what software development is. Maybe that’s what people talk about when they say they don’t “have a mind for programming”. It’s not that they can’t, but they’re not the kind of person who feels the need or desire to question everything, get their hands dirty, and learn about as abstract a concept as an arbitrary piece of code. Personally, I find it a lot of fun. And you sort of do this “rolling around” in levels, going from the simplest to the hardest.

At the simplest, you’re just trying to acknowledge what it is you’re looking at. If it’s a Rubik’s Cube, that’s what you’re trying to come up with. If it’s an encoding issue on a web page, that’s all you’re trying to identify. Then you start fiddling with it. With our Rubik’s Cube example, start twisting things. Start small; make sure you can find your way back. If it’s a piece of text that’s wrong, maybe find where that text comes from, or how it’s being displayed.

All along the way, you’re looking for clues to figure out what you ultimately are trying to get to. If it’s a Cube, maybe you’re trying to solve it. If it’s a bug, maybe you’re trying to find the root cause and possibly fix it.

For your cube, maybe you start trying to solve a side. You draw upon your knowledge of geometry, group theory (or at least the intuitions you have about it), your memory of fiddling with sides and your goal to try to move all the pieces to one side. For your software bug, maybe you open the code that generates that text, and you start substituting other pieces of text seeing if they yield the same bug.

This process can go as deep as it needs to. For the Rubik’s Cube, entire branches of mathematics are in no small part devoted to solving puzzles like the Rubik’s Cube. For software, there are web tools, open source tools, tools by Microsoft or Apache to solve these tools. There are test frameworks and troubleshooting methodologies. All of which will come with practice and experience. Many of which you won’t need for all problems, but are tools none the less at your disposal.

The point is, when you see an intractable problem in front of you, don’t be scared. Pick it up and start playing with it.

Tuesday, January 31, 2017

SQL ISNUMERIC() function

Imagine you have a column of data which is supposed to contain numbers, but you don't know which ones are numbers. Imaging you also live in a world where SQL Server 2012 and above don't exist. Now imagine you stumble across the function ISNUMERIC one day and think to yourself "huzzah! My worries are over!". Now imagine you put that check in to your code, push it to production, and low and behold, it doesn't do what you expect.

This is documented elsewhere as well but ISNUMERIC might not do what you expect. It includes characters which aren't strictly speaking numeric values. For instance, it contains several currency signs. It allows some white space characters, and it allows commas, pluses, minuses and more. Consider the following.

    Idx = num,
    Chr = char(num),
    IsNum = isnumeric(char(num)),
    IsNum_e0 = isnumeric(char(num) + 'e0')
from (select top 255 num = row_number() over (order by (select null))
      from sys.all_objects) a
order by isnumeric(char(num)) desc, num 
You can see all the characters (from the first 255 ascii characters) which it will consider to be "numeric". It's also worth noting that this is not a comprehensive list of characters though, as I'll illustrate in a way which improves the functionality of isnumeric in many cases. the trick is to add 'e0' to the end of the string you're converting. So 123 becomes 123e0. Since this is scientific notation, it's considered a numeric value; and rightfully so. but "$\e0" is decidedly NOT numeric. In the statement above, I included a "e0" suffixed column as well as you can see it cuts out everything that's not a number.

So remember ISNUMERIC is not quite what you think, but if you're expecting a run-of-the-mill integer, float, or decimal value, appending "e0" to the end can get you a long way.