Database Design is Easy — Or Is It?

I came across a Request for Proposal from a government entity whose sole purpose was to conform to the data collection standards of a superior government entity.   The following entry is paraphrased for brevity.

Yes/No Questions. The “yes” response and the “no” response each has its own column in the spreadsheet tool.   To specify “yes” for a given question, the user should enter a “Y” in the given question’s “yes” column. To specify “no” for a given question, the user should enter an “N” in the given question’s “no” column.  One question on the form is an exception to this “standard” where there is only one column in the spreadsheet tool for this question, and users should enter a “Y” for “yes”  and “N” for “no”.

This was written by professionals who should be in a different profession.  I struggle with how to explain why this is fubar.  So here goes.

First of all, these are questions on a form, questions, mind you, with only two answers, “yes” or “no”.  Really.  How complicated can that be?  Apparently, very complicated.  If two responses (one in the Yes column and the other in the No column) are allowed, how will one know which response reflects the facts of the respondent?

The noted “exception” should actually be the “standard”.   Most likely, this was written by lawyers to allow for flexible interpretation of the law when enforcing compliance so as not to hurt anyone’s feelings.  Or maybe it exists to confuse the respondent, so they will give up filling out the form entirely to seek out other sources of assistance.  Whatever the reason, the person designing such forms should be in a different line of work because either their brain or their heart are not in it for the right reason.

Why is this important?  Say you wanted to report on the number of respondents who replied to a particular question.  Count the ones who respond “yes”.  Count the ones who respond “no”.  Simple, right?  OK.   Now calculate the percentage of respondents for each.  Well, if some put down “Y” in the Yes column AND “N” in the No column, the sum of the counts of each would be more than the total number of respondents.

Now, image you are a legislative leader and you are reading a compliance report based on these statistics in order to place a vote for a multi-billion dollar extension of the program.  You might see that 65% said “Yes” and 63% said “No” to a particular question on which you would like to base your decision.   Or maybe, the data analyst preparing the “required pie chart” report will recognize the data problem, and fudge the responses so the chart reflects HIS political preferences.   Democracy in action.

I have all the answers, its knowing the right the questions to ask that is the challenge.

For more examples of this kind of thing, please visit my favorite website,

Bluetooth Beacons in Retail and Howler gave a webinar on the state of the market for Bluetooth beacons.  Howler said hundreds of Beacons have been deployed for city-scape retailers in Boston, Las Vegas, and San Diego.  They reported over 50% conversion rates for customer’s walking by responding to real time notifications.  The user has to install the Howler app on their smart phone so it can read the Beacons, but they are not retailer-specific, so the medium seems to be an attractive channel for advertisers in boutique retail and restaurants.  Howler advertisements can include a URL to drive traffic from the alert to other content.

Driving responses to mobile ads triggered by physical presence to URL content creates the need for real time content management to offer specials, coupons, and chalk-board menu adjustments.  The K-Mart “Blue-Light” special just got upgraded! A new generation will know it as the “Blue-Tooth” special!

Konduit can manage URL-accessed content in real time.

Purchasing Software Without Testing It

So I came across this listing for a freelancer:

I already have purchased a complete POS software with source code built in .NET that uses Access for database. The POS software has 3 programs which are fully working. The server, client and PDA software. The software is fully functional but is not very user friendly and does not have proper reporting features, it can not complete a sale etc..

This person is really fooling himself.  A Point of Sale (POS) software package that “cannot complete a sale” would not be very usable for anything.  Perhaps it just needs some configuration changes, but if you didn’t buy that software with professional services of the person who developed it, then there is not much another developer will be able to do.  Unless you want to pay them to learn the source code and then strive to keep their interest without paying them full time, you will be soon out of luck (see “I Want to Own…” article).

Packaged software solutions used to run your unique business should include professional services to install and configure it.  You can read the manual of course.  But that is the same with anything.  You hire professionals to install new electrical or plumbing fixtures.  Why try to build and maintain software for running your business?

Hire a professional.  The results will be much more satisfactory.

My Data Loads Take Too Long

During twenty years of creating relational database applications for clients, I have optimized data loads primarily by changing INSERT statements into a BULK LOAD process.  In SQL Server, an external command line utility (bcp.exe) allows bidirectional data streams to flow.  In MySQL, the inward flow is managed by the LOAD DATA INFILE command.

The key to using these BULK LOAD processes is formatting the data properly in files before loading the data.  The best language for parsing large streams of data is Perl.  If you convert your INSERT  statement-generating PHP, Java, C#, VB, etc. to a Perl scrubbing and formatting operation and use the database’s BULK LOAD utility, I am sure you will see a significant increase in the performance of your data loads.  I have used operating system shell scripting languages to tie it all together for automation.

Another factor is the configuration settings of the relational database logging operation.  Most simplified databases do not log transactions because they do not enforce ACID principals.  In fact, most databases, relational or not, are based on straight-forward ISAM principals.  Transactional databases, those that require recovery of transactional (e.g. INSERT) failures, perform extra operations to allow for that recovery.   But doing this at every INSERT of a row slows the load process down unnecessarily.  In a data warehouse, this transactional logging can be turned off.  Data recovery is reduced to the size of each bulk load operation, but then that is what you are already managing with your load process.

A BULK LOAD operation in transactional databases will treat the entire load as a single transaction, so breaking up a large set of rows into separate files may make the processes more manageable.  The cause of failed file loads can be investigated and dealt with on a more granular level if the files are split into manageable pieces.  What defines “manageable” is determined on a case by case basis by the latency and use requirements of the data.



Computer Science Degree — Is It Worth It?

Here is a request for a proposal taken from a popular freelancing site (spelling errors included):

I am looking for someone who has good experience writing thesis projects. I have a deadline of December 10th to submit my thesis. I am a Computer Science student and almost at the end of my completion of the degree. I have to present a good strong thesis project related to IT which has some programming or implementation involved in it. Below is the brief description which my professor has given me to write my thesis. I am looking for someone who should be good at writing strong tehnical thesis. Money woudl not be an issue for the right person.

I come across these occasionally and wonder what will this person do when they enter the “real world”.  This one is particularly telling with its directness.  Will an employer be able to detect their lack of ability?  Will they be fast-tracked to management because of their pedigree?  Will their spell-checker prevent future exposure to their incompetence?

I believe, for the most part,  there are good schools producing good students.  But seeing things like this worry me.  The general lack of enthusiasm of Millennials and the misplaced emphasis on job placement among colleges (and universities) has degraded our educational system.  And, assuming this is a foreign college, I have much doubt about our ability to fulfill the vacant computer programming positions by increasing the H1B visa quotas.


Platform (PaaS) , not Infrastruture (IaaS), as a Service

Jonathon Feldman wrote in February “Why IaaS Won’t Happen in Most Enterprises“.   He makes the argument that departmental business needs will drive spending on Platform as a Service provisioning.   The argument is a solid one that Konduit has been using to satisfy end-user needs for over ten years.

Like software application development tool vendors, most PaaS providers rely upon a following of application developers to train and certify in their technology, all at significant time and dollar investment.  If they attract enough followers, then major tool vendors eye them for acquisition, which is followed by migration or deprecation, leaving customers to start over again.

When PaaS licensing is combined with application development services so that customers are not left with unsupported mash ups, we call that Support.  The development, deployment, and support of applications written using Konduit’s Transaction Editor Platform as a Service stays within the control of Konduit engineers.  A customer’s subject matter experts remain in control of the work flow and practices that define the software’s functionality, without having to understand the mechanics underlying the computer logic, security, reliability, or presentation methodologies.  They just get to use it and share it as needed.  In essence, Support as a Service, provides infrastructure, platform, and services to the end-user without requiring their understanding of the technology.  Unfortunately the acronym SaaS has already been taken.  Please comment with your ideas for a new acronym for Support as a Service.


Cloud Economies of Scale – Not!

Over the past five years, software vendors have been moving their customer-premise offerings to cloud services.  The cost of software license fees are added to costs for hardware, network, and security infrastructure.  Millions of dollars are being invested in putting packaged software in the cloud so customers can expense rather than amortize their information technology purchases.  Like the record industry, vendors create bundles (i.e. albums) in order to stack less-used features with their core features.  Monthly subscription service rates are calculated like lease agreements, so accountants can depreciate these investments.  The assumption is that economies of scale apply to technology.

These investments ignore the fact that technology is distributing workloads, not consolidating them. The Internet of Things is adding points of input.  Arrays of commodity smartphone components continue to replace multi-million dollar engineered products from digital microscopes  to satellites and cable TV.  The Internet is still made up of mostly personal computer technology all wired together.  Blade server technology uses a distributed design to imbue redundancy, and thereby reliability, into their products.  Distributed computing is the only possible way to solve the most difficult problems.   Unfortunately, software design has not kept pace.


The act of creating “packaged” or “common off-the-shelf” (aka “COTS”) software products requires accommodating the needs of a diverse set of users.  This results in a small (20%) subset of features that remain useful for any particular group of users, while the majority of features (80%) remain unused. Custom software, by contrast,  contains only those features that users deem useful.


The development and support of custom software requires knowledge of how business processes operate.  Small enterprises that cannot afford a full-time programmer or have difficulty retaining staff due to volatile business cycles, need a software solution that applies common computing principals while remaining flexible in the data elements it can handle.  Spending millions of dollars customizing an enterprise package is out of reach for most small enterprises.  (Large enterprises just waste a lot of time pretending to use more features.)


Most small enterprises rely upon spreadsheets and word processors to digitize their paper document world.  Vendors reply with file-based solutions to help them “organize” their digitized documents.  Few vendors offer structured data solutions that enable searching, sorting, and sharing of the transactional details of documents.  Companies that combine the ease of use of a spreadsheet with the accountability of a database using a common business process methodology can solve this problem.   A common business process methodology enables to highly customize web-based applications for each client, while retaining a secure, cost-effective operational environment.


Scalability of the cloud is not achieved by moving bloated software to mainframe hardware.  Cloud scalability is achieved by distributing only “needed” features and applying a common business process to a unique set of data items. This allows software and hardware to be shared while content remains specific and relevant.


What is a database service?

Database as a Service (DBaaS) may sound like a service business, but really its just another specialized Infrastructure as a Service (IaaS). The CTO Vision article on DBaaS talks about hosting a database management system, but not hosting the application itself. Specifically, the author states “I believe that the reason for the draw is that databases, while critical to your applications, are generally a bit difficult to configure, scale and operate correctly. When presented with the opportunity to off-load that work to a provider, developers jump at the chance.” Developers seek help with databases because they were taught little in school about them and their experience becomes clouded by the many misperceptions about relational databases best described by Fabian Pascal. A good starter article is “Data Management’s Misconceptions” by Joab Jackson.

A software application that relies on mash-ups of packaged products without professional services to clarify and organize the meaning of data is, well, meaning-LESS. Database management systems and programming languages cannot, by themselves, provide the meaning, integrity, and consistency that gives an application purpose and usefulness. Some ONE needs to be facilitating the discussion of what functionality is needed, and to clarify what the nouns and verbs used to describe a business process mean. Without this clarity of purpose, any software application will eventually be doomed. combines the professional services needed with a process framework that accelerates the iterative cycle of process adaptation to empower young businesses with a virtual “IT Department” where none exists. This service is delivered over an encrypted connection on the Internet, requires little upfront cost, no contracts, and a predictable recurring monthly fee structured to reflect the complexity of the application delivered.

The term “database” is filled with mystery, but in reality it’s just about communication and logical organization, something every business needs in order to thrive. The more organized your business information is, the greater chance you will have to settle misunderstandings with your customers and vendors. If you can audit the history of data changes, you can identify and remedy mistakes and know who needs more training.  Database management systems provide a critical part of what is needed.  Information technology is more than hardware and software as commodities. Information technology requires people who understand your business and can rapidly adapt your software-based process automations to changes in the marketplace or operations.  To be effective, a software developer must understand how to use a database management system in the context of a business process automation application.

This is why companies are still afraid of the cloud

This is why companies are still afraid of the cloud


On April 25, a systems administrator was sentenced to 33 months in prison for intentionally causing damage to a protected computer. Jonathan Hartwell Wolberg of Tucson, Ariz., will end his prison term with 36 months of supervised release for sabotaging his former employer’s cloud computing server.
According to court documents, Wolberg had worked as a systems administrator for “Company A,” a cloud provider headquartered in Virginia. After resigning, Wolberg continued to enter the Company A cloud to damage its servers, its reputation, and its business. This included shutting down “key data servers,” including those supporting hospitals. As a result, Wolberg caused hundreds of thousands of dollars in damage.
This is the kind of story that scares — or is used to scare — IT organizations about the cloud. After all, core to the cloud-client relationship is your trust in the client provider to keep hackers away from the systems carrying your data and running your services.
Wolberg committed the crime, but Company A made it all too easy for him to do so.
They’re rare exceptions, fortunately, especially at the major cloud providers.
That said, it’s a good idea to ask your cloud provider a lot of questions, especially if it’s a smaller provider. In some cases, a premigration security audit is in order. The provider should be receptive to such validation requests — an enterprise looking to use its cloud could result in a business relationship that lasts for years.

Read the full story at