Search volume for Open Source, Google, Microsoft, and Apple

(Credit: Google Trends)

On this blog, MySQL and Ubuntu make an appearance in the top-25 most read stories, but Microsoft, Apple, and Google dominate the most-read stories, despite constituting a relatively small number of my total posts.

I note this data only to remind everyone, myself included, to take ourselves a little less seriously. It's not that open source isn't critically important, because it is: It is the heart of computing going forward. But our petty controversies are just that: Petty. The stakes are pretty small, given that open source is being woven into the fabric of software's future, even within these giant software vendors, regardless of our squabbles.

Do you know which open source feature is the most important? Do you know which open source database rocks and which one sucks? Is MySQL better than Postgres? Is Ingres worth considering? How does Firebird compare? Have you used, or have you considered using, an open source database?

Take a survey. It's only 15 questions so it takes just a few minutes.

I'll post a link where you can get the results once they have been compiled and prepared.

BTW, this isn't my survey. I'm just passing on the link.

LewisC

1. Status page now reports daily error code summarizations, connection failures to hosts
2. Talkback report page now reports on myback_talkback script version number
3. Talkback report page now tracks the my.cnf file during the backup process, and the cnf file contents for each host are stored as LONGTEXT in the monitor database
4. Multi-tier access levels, admins (rw) and general users (ro)
5. View tables for the status page summarizations speed up reporting
6. Daily summary email - similar to the status page, instead of singular emails for each backup success/fail notification

Look for 1.3 being released very soon! You’ll be able to find it here: http://sourceforge.net/projects/monolith-mysql

My interview with Tom and Charlie (14:19)  Listen (Mp3)   Listen (ogg)

Charlie Nutter and Tom Enebo -- the JRuby dyanmic duo (and wearing the shirts to prove it).

Some of the topics we tackle:

• A historical background, Ruby, then Rails, then JRuby.
• What the guys were doing pre-Sun and how Tim Bray lured them to the fiery orb.
• Three licenses is better than one (GPL, CPL, LGPL)
  
• How JRuby is working with GlassFish and NetBeans.
• The DaVinci machine project, jMaki and what ever happened to AJAX?
  
• Might we see a JPerl?
• Looking to get more involved with MySQL

The question itself might be innocuous but Matt’s use of the term “free-riders” prompted a couple of angry responses. Storm in a tea-cup stuff really.

Meanwhile, in a unrelated post, Savio Rodrigues was wondering “is the community hurting the OSS business model?” The answer to that might well be “it depends which OSS business model you’re talking about” but nevertheless Savio’s point is that community reaction to MySQL’s commercial plans may have undermined not MySQL’s business model and development plans.

“This will help proprietary vendors maintain the feature/function gap vs. OSS vendors. Recall that for the majority of single-vendor backed OSS products, there is virtually no cost savings vs. developing closed-source software. To close the feature/function gap, OSS vendors need faster revenue growth to fund this development expense,” he writes.

“The OSS vendor community needs leaders who will stand up to ‘the community’ and make the tough business decisions needed to ensure that OSS isn?t relegated to a small revenue slice of the software industry pie.”

In the light of the reaction to his post, Matt Asay responds to this suggestion: “Most days I’d find this simply wrong, but reading the responses to a harmless suggestion that people should contribute more to open-source projects…it makes you wonder.”

I have previously observed a growing animosity of some sectors of the open source software user community towards commercial software vendors and activities. Where once the commercial success of an open source vendor was to be celebrated, increasingly it seems it is treated by some as a reason for caution and doubt.

I may be wrong, but there appears to me to be a strengthening commitment in some quarters to the ideals of the Free Software Foundation in rejection of the commercial opportunities provided by the Open Source Initiative.

What do people think? Coincidence, or are strange things afoot at the Circle K?

Recently, I ran into cases where the developers have decided to use memcached over MySQL style architecture after reading some/many good/nice articles about it without giving a damn to their requirements. I would like to list few things as a checklist for developers to decide on their architecture. There is still no precise answer but sometimes few cases can be just ruled out :).

1. What is the total size of your data? It might be a possibility that you can keep the data in memory in each node, or MySQL can just keep the whole thing (data+indexes) in a buffer.
2. How frequently your data is updated? Very frequent updates may lead to low cache hit ratio for memcached data. And refreshing memcached too many times may lead to unnecessary overhead. Remember doing [get,get,set] vs [get].
3. What is the peak load on your system? Consider if MySQL itself can handle the peak load or otherwise if even memcached cannot handle the peak load with given infrastructure.

I generally ask people a simple question, Why they think they should be using memcached (or something else even)? To shock them, I even ask them "Why they think they should be using MySQL?". And believe me, this is what I believe developers should be asking themselves.

There is only one good argument against this, what if tomorrow you need to suddenly scale or what if your projections need memcached? In such cases, I suggest people to design their data layers in a flexible way, flexible enough to allow things in and out.

“The Sun Fire X4240, powered by the AMD Opteron 2200 and 2300 processor series, is a two-socket, 8-core, 2RU system with up to twice the memory and storage capacity of any system in its class. It’s the first and only two-socket AMD Opteron system with sixteen hard drive slots in a 2RU form factor.”

Well, now that I work for Sun, it ends up being a shameless ad and boasting :) But back when I saw information about this product, I wasn’t my first thought was “wow, thats the best machine for scaling up scaled out environments!”.

In web database world people agree that number of spindles (disks!) matters - remember YouTube’s “think disks, not servers” mantra said during the scaling panel at MySQL conference. Before, getting such number of spindles would’ve required external arrays taking space and sucking power (TCO! ;-)

And for us… it probably means we can finally start doing RAID10, instead of RAID0. :-)

By the way, that box even has Quad-Core service processor. Way to go! :)

MySQL is Free Software, licensed under the GNU GPL. People frequently use the two phrases “Free Software” and “Open Source Software” as synonyms, but there are very large, very important differences.

The difference between Free and Open Source

Open Source is much more of a development methodology than a philosophical standpoint. The first thing on the Open Source Initiative’s website is this introduction:

Open source is a development method for software that harnesses the power of distributed peer review and transparency of process. The promise of open source is better quality, higher reliability, more flexibility, lower cost, and an end to predatory vendor lock-in.

In contrast, Free Software is not about development practices at all. You can develop Free Software any way you like; what makes it Free is the license. Free Software is about protection of rights and freedoms. It is a moral and ethical platform. The promise of Free Software is quite different from “better quality, higher reliability…” Free Software is not about quality or reliability.

So why is MySQL not Open Source? Simple. Sun/MySQL uses a closed development model. Nobody can get code in from the outside without accepting a Contributor License Agreement (CLA) which requires surrendering important rights, including ownership of the code. Sun/MySQL controls the code absolutely and maintains ownership of it. And even people who have signed the CLA report their patches stagnating — often for years — and still not being accepted into the source. This is not Open Source.

Open Source software is usually maintained, owned, and controlled by a decentralized network of peers. This is exactly the opposite of MySQL. You cannot get more opposite. The differences are often summarized as the cathedral versus the bazaar. I’m not sure this analogy always holds or is always useful and accurate, but it’s a helpful piece of common vocabulary.

Why this matters

This matters because both Freedom and an open development model are necessary to an empowered, enlightened, free society. Licensing isn’t the only thing that matters: ownership matters, too. So does control.

Google’s patches to MySQL are a good example of excellent code with many simple, highly useful features that have not been included into the official MySQL distribution. And there are no signs of that changing, as far as I can see.

I’m not the only one who notices this. Here’s another quote:

With all due respect to Marten, there is a significant difference between the captive open source development model for MySQL and the community open source development model for PostgreSQL.

If this interests you, perhaps you would like to join the discussion on the oursql-sources Google group.

My first thought was that the customer had deleted from the table, which leaves "holes" in the middle of it and prevents concurrent inserts. (You can configure the server to permit concurrent inserts even when there are holes, but it's disabled by default.) However, that turned out not to be the cause; the table was only inserted into (and selected from). Instead, the blocked statements were because of INSERT... SELECT statements that were running against the table, selecting data from it and inserting into another table.

Let's look at what happens here: suppose you have two tables tbl1 and tbl2 and concurrent inserts into tbl2 are running fine. If you now run the following query,

The concurrent inserts into tbl2 can block. This happens if you have the binary log enabled. If you think about it, this makes sense and is correct behavior. The statements have to be serialized for the binary log; otherwise replaying the binary log can result in a different order of execution.

The MySQL manual actually says this, but not in the clearest way. It just says

If you are using the binary log, concurrent inserts are converted to normal inserts for CREATE ... SELECT or INSERT ... SELECT statements.

If you use mysqladmin debug, you'll see an ordinary SELECT gets a lock on the table like this:

But on INSERT...SELECT, you'll see this:

That read lock is what's blocking the concurrent inserts from happening.

There's no solution to this, if you need the binary log enabled. (It needs to be enabled for replication.) There are workarounds, though. You can use the old trick of SELECT INTO OUTFILE followed by LOAD DATA INFILE. You can use InnoDB instead. Or you can do something more elaborate and application-specific, but that's a topic for another post.

Entry posted by Baron Schwartz | 3 comments

Add to: | | | |

Ok. So I just looked at two blog posts about MySQL log rotation. While not technically incorrect, I’d like to disagree with the final answer, mainly because recreating the wheel drives me crazy.

Rather than writing a custom script to do this and putting that script in cron, why not use logrotate?

We already ship fully functional log rotate scripts in the Debian MySQL packages. Here’s an example:

/var/log/mysql.log /var/log/mysql/mysql.log /var/log/mysql/mysql-slow.log {
        daily
        rotate 7
        missingok
        create 640 mysql adm
        compress
        sharedscripts
        postrotate
                test -x /usr/bin/mysqladmin || exit 0

                # If this fails, check debian.conf!
                export HOME=/etc/mysql/my.cnf
                MYADMIN="/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf"
                if [ -z "`$MYADMIN ping 2>/dev/null`" ]; then
                  # Really no mysqld or rather a missing debian-sys-maint user?
                  # If this occurs and is not a error please report a bug.
                  if ps cax | grep -q mysqld; then
                    exit 1
                  fi
                else
                  $MYADMIN flush-logs
                fi
        endscript
}

This does the re-creation, flush logs and handles failure conditions gracefully. It also compresses old files. Even better - your system already has logrotate running (unless you run a really bad distro)

Now, if you aren’t on Debian, don’t just copy this verbatim, as you probably don’t have a debian-sys-maint user or an /etc/mysql/debian.cnf file. In Debian, we make this at package install time and refresh it on upgrades. It’s a system user with an autogenerated password put in a file only readable by root, that allows system scripts like this to run unattended.

Once you read up a little on logrotate, you’ll realize it’s really great at just about anything you need to do in the log rotation field. Have fun!

A month or so ago, I was having a conversation with The 451 Group’s Vice President of Research Services, Simon Carruthers, about ways to expand the offerings of the CAOS Research Service. The CAOS Research Service includes aspects that are public (such as this blog), but the majority of the work that we do is accessible only to our paying clients, namely our research articles and reports, as well as our advisory service.

We made the decision to add a biweekly conference call for clients - a roundup of the top open source issues for the past two weeks, with insight provided by the 451’s open source analysts, Jay Lyman, Matt Aslett, and myself. For those who could not dial in for the call, we provided the audio file as a private podcast. Last Friday was the second conference call. The concept was inspired by Diane Rehm’s News Roundup on NPR.

As of today, we have turned the client conference call into a public podcast. This comes at the request of our clients, actually. Thanks to Simon for the go-ahead to make this open to all. Clients will be allowed to dial-in to the live call and submit questions in advance.

You can subscribe to the CAOS Theory biweekly podcast via iTunes or RSS feed. We’ve posted the first two ‘episodes’ and the next one is scheduled for Friday, May 23rd.

2008.05.09 Agenda:

• Black Duck Software acquires Koders
• Sun, IONA, Sourcefire, and Actuate announce quarterly results
• Sun’s new OpenSolaris push
• Adobe and the Open Screen Project
• SpringSource’s new platform

2008.04.25 Agenda:

• The state of open source venture funding
• What’s up with MySQL?
• Desktop Linux troubles
• Qlusters stepping away from open source
• OSI Board election
• Open source social - centralized or distributed?
• The Open Source Census

We’d love to get your feedback on the podcast - post a comment here or send me an email - raven.zachary@the451group.com. Thanks!

http://themattreid.com/wordpress/?p=34

Although the panel itself wasn't recorded, immediately after it concluded, the five of us headed to the make-shift podcast studio we had set up at the event and recorded the following discussion.  Listen at your peril!

My Discussion with the panel (44:08)  Listen (Mp3)   Listen (ogg)

No glasses? No facial hair? Get in the back.
Zonker Brockmeier, OpenSUSE; Jono Bacon, Ubuntu; Karsten Wade, Fedora;  Glynn Foster (standing), OpenSolaris 

Some of the topics we tackle:

• How the gentlemen balance the needs of the community with the needs of their corporate parent.
• Working with upstream providers.  Is the heat Ubuntu gets on this subject justified? Karsten's mantra (Glynn, a former GNOME director, also offers a perspective from the upstream)
• Shout-outs to Debian
  
• Governance and how each of the distros handles it (the community/corporate split and who picks the members)
• In the last third of the podcast we head off in to the realm of the philosophical and address Free vs. Non-Free software, the "Vocal Minority," Richard Stallman, mini-Stallman's and all things holy.

http://dev.mysql.com/doc/refman/5.1/en/insert-on-duplicate.html

A common usage pattern for this is “lazy initialization“ of a row in a database, which is exactly what my team was using it for yesterday to solve a problem in the backend for version 2.0 of the MySQL Enterprise Monitor. However, we ran into an issue where Hibernate would throw an exception complaining that when the INSERT was turned into an UPDATE, it couldn‘t retrieve the generated primary key value (we are using auto increments on this particular table, as it‘s not a high insertion-rate table).

To understand why this happens, you have to know a little bit about how Statement.getGeneratedKeys() works with MySQL‘s JDBC driver. When an auto increment value is generated by MySQL, that value is returned on the wire as part of the message back to the client that contains the update count, warnings, etc. When an application asks for the generated keys from MySQL‘s JDBC driver, it takes the value returned on the wire and crafts a “synthetic“ result set to represent it.

In the case where “ON DUPLICATE KEY UPDATE“ is in play, it turns out that the value returned on the wire, by default is indeterminate, which we would‘ve learned from reading our own manual more carefully:

“If a table contains an AUTO_INCREMENT column and INSERT … UPDATE inserts a row, the LAST_INSERT_ID() function returns the AUTO_INCREMENT value. If the statement updates a row instead, LAST_INSERT_ID() is not meaningful.”

It turns out, the answer to this is to add a little assignment to the UPDATE clause:

INSERT INTO table (a,b,c) VALUES (1,2,3)
ON DUPLICATE KEY UPDATE id=LAST_INSERT_ID(id)

With that little “id=LAST_INSERT_ID(id)” bit added to the INSERT statement, the server now returns the already existing auto increment value back to the client, and Hibernate is happy again.

I will post the slides a little later today after the PLUG meeting. I'm really looking forward to meeting Ryan and the other folks from PLUG and getting a short tour of Omniture, a large web analytics company that uses MySQL (no, it's not Overture. Totally different. One of those is now extinct.)

At Wikimedia we have one database server which replicates from multiple (like 15!) masters. It even splits replication streams by database, and applying changes in parallel.

All this stuff is done by external replication agent, Trainwreck. It is public-domain software, which was written by River, doesn’t have much documentation, works only on Solaris (River likes Solaris), unless you comment out all process management blocks, which use doors and other Solaris specific API.

It lives in Wikimedia SVN, and can be checked out using:

svn co http://svn.wikimedia.org/svnroot/mediawiki/trunk/tools/trainwreck/

It sits there, maintained just for needs of that specific single server (ok, there might be two or three), so if anyone wants to make it available for broader audience, feel free to fork a project to some community-oriented place, add all nice features you need. :)

MySQL is back on Open Source track and that is definitely the best news for all (including community, MySQL and Sun as well). I think that now Sun/MySQL have agreed to the importance of community, it becomes community's responsibility to give them more reasons to believe so. Let's participate like never before.

Kaj, in his post says "...model to be useful for both those who spend money to save time, and those who spend time to save money". This is what Open Source is, isn't it?

All in all, a decision most awaited and most welcome is taken. Thanks a ton to everybody who made this happen.

Just briefly: Why do we even care for prepared statements? For stateless web applications the benefits are mainly protection against SQL injection and better readability. In some rare cases in theory also better performance if the same statement is executed multiple times in the same request. The disadvantage is that for most web applications queries are rarely executed more than once and therefore the separation of parsing/planning and execution just means that there is an additional roundtrip to the database. Also in very long running database sessions the prepared query plan could get outdated, which could also spell disaster for performance. or in other words: prepared statements are no silver bullet.

The problem described in the above blog post is of course since the idea behind prepared statements is to separate the parsing and planning stage from the actual data that is to be used in the WHERE clause of the query. Modern cost based optimizers keep detailed statistics about data distribution, which is difficult to leverage if there is no data available during the planning stage. This blog post however focuses on another aspect that is not yet relevant to MySQL, since there is no support for partial indexes in MySQL just yet. However RDBMS like PostgreSQL do support partial indexes. Again since the data is missing during the planning stage, the database is unable to determine if an existing partial index matches.

So what can be done to fix this if we do want to stick with prepared statements? On the user side one thing one can do is move the relevant filtering data into the prepared statement itself. This of course means that you loose the benefit of the prepared statement for the relevant pieces of data, but it might be an acceptable balance. Of course if this data contains user supplied data, you now need to manually make sure that the data is safe.

The other approach is simply switching to emulated prepared statements, which most database layers support (like PDO). This also fixes the issue of adding a second round trip to the database for every execution. However it means you put yourself at the mercy of the parser embedded in the database layer you are using. Actually in the case of PDO there is little chance to get around that anyways. Worse yet, the PDO parser is known to be buggy by design.

On the database one approach that has been taken by Oracle since I think 9i is to delay the planning stage until the first execution. The query plan is then optimized for the first data set. I do not know how this works in detail, as I presume that if the plan uses a partial index as a covering index and the next data set is not contained in the partial index, that this approach would fail. However it gets the job done for people that just want to use prepared statements for one shot execution to get SQL injection protection and better readability.

Another approach that I do not think any RDBMS has taken (probably because its either hard or not feasible .. or both) is to keep multiple query plans around. In this scenario in the initial planning phase the database would also store some simple parameters that define if an existing query plan will match or not. If the plan does not, then a new plan is generated, while the old plan is kept around. This way a single prepared statement could eventually keep multiple query plans around. This would also solve the issue with partial indexes being used as covering indexes.

Yet another approach, which would be less interesting for short running sessions as are common in PHP applications, would be that if the query is executed and the performance is slower than expected, that this then triggers a replanning. Potentially also keeping the old plan for old data sets. This would also solve the issue of long running sessions having out of date plans.

Overall it seems for stateless web applications the best approach is using a database layer with good prepared statement emulation support. I hope that PDO, will one day fit this criteria. Until then the MDB2 parser seems to do a much better job. Though MDB2 is of course implemented in PHP, which carries its own performance overhead and is not E_STRICT compliant due to the PHP4 compatibility.

I am extremely happy to hear these news ! This is good for MySQL as a company, MySQL customers and MySQL users.

I'm hoping Community feedback was serious contributer to this decision, though I know there were a lot of Internal discussions as well. In any case this sends a great message to community - Speak up and you may be heard.

I also hope Marten Mickos took this decision being convinced rather than getting the order from the top as this is only one battle in "what is going to be opensource" war

Anyway thank you everyone who made this happen, in particularly Monty, which I know fought a lot for this.

P.S. This is great news but I'd like to see and know more. Looks like servers is left alone being Open Source, what is about MySQL WorkBench, MySQL Proxy Extensions, MySQL Monitoring software ?

Entry posted by peter | 2 comments

Add to: | | | |