-
SpyMemcached Transcoder with PHP PDO Objects using ZLIB
My technology stack services more then 2 Million Daily Active users. Its very basic. PHP talks to mySQL, Memcache, RabbitMQ, Gearman and Facebook. Now that we have more Java specifically to support our SmartFox Server and other services, blurring the lines between what data is set in PHP and what data is read in Java is very necessary.Java J-Connect makes reading mySQL Data as simple IMHO as PHP's PDO. What is hard is reading PHP's serialized format from PHP's Memcache library.In PHP there are two main C backed Libraries. There is Memcache the original PHP library which I happen to use, and Memcached which is the library I wanted to use but didn't deploy since EC2 package system conflicted and cause issues (I fixed them but to late to deploy). Memcache stores data in PHP's serialized format and compresses it via ZLIB, while Memcached can store data as PHP's serialized format, JSON, Binary Serialized (which is rather awesome), JSON Array Notation and has a multitude of compressing formats none of which are pure ZLIB that I noticed.Here is the problem. Spymemcached is a lib for talking to memcache but can't unserialized PHP serialized format (or read it natively and return a string) and cannot Decompress ZLIB but can Decompress GZIP. Now a great speed up would be to use PHP's serialized data set from PHP and share memcache resources from PHP and Java just like what is done for the mySQL resources.What needs to be done? Well, build your own Transcoder for Spymemcached. Fortunately Spymemcached documented an interface to do just that.What is needed. Implement the spymemcached Interface defined here. Use org.lorecraft.phparser to unserialize PHP data defined here. Return the Object. Below is the code.package com.schoolfeed.spymemcached;import java.io.ByteArrayInputStream;import java.io.ByteArrayOutputStream;import java.io.IOException;import java.util.zip.InflaterInputStream;import net.spy.memcached.CachedData;import net.spy.memcached.compat.CloseUtil;import net.spy.memcached.transcoders.BaseSerializingTranscoder;import net.spy.memcached.transcoders.Transcoder;import org.lorecraft.phparser.*;public class PHPSerializedTranscoder extends BaseSerializingTranscoder implements Transcoder <Object> { static final int COMPRESSED=2; /** * Get a serializing transcoder with the default max data size. */ public PHPSerializedTranscoder() { this(CachedData.MAX_SIZE); } /** * Get a serializing transcoder that specifies the max data size. */ public PHPSerializedTranscoder(int max) { super(max); } /** * decode the byte data from Memcache decompress it if necessary and return the Object * @param CacheData - the byte data is turned into a object * @returns Object */ public Object decode(CachedData d){ byte[] data=d.getData(); Object rv=null; String ds="N;"; if((d.getFlags() & COMPRESSED) != 0) { getLogger().debug("Looks like d is compressed"); data=decompress(d.getData()); } ds=decodeString(data); getLogger().debug("DECODED: [" + ds + "] about to SerializedPhpParser"); SerializedPhpParser sp = new SerializedPhpParser(ds); try { rv = sp.parse(); getLogger().debug("Parse was cool!!"); } catch(Exception e){ getLogger().debug("Not a PHP Object? : " + ds); rv = ds; } return rv; } /** * PHP Memcache stores compress data in ZLIB format override the base class decompress method to handle ZLIB * * @param byte array - raw data from Memcache * @returns byte array */ protected byte[] decompress(byte[] in) { ByteArrayOutputStream bos=null; final int BUFFER = 2048; if(in != null) { ByteArrayInputStream bis=new ByteArrayInputStream(in); bos=new ByteArrayOutputStream(); InflaterInputStream iis = null; try { iis = new InflaterInputStream(bis); byte[] buf=new byte[BUFFER]; int r=-1; while((r=iis.read(buf, 0, BUFFER)) > 0) { bos.write(buf, 0, r); } } catch (IOException e) { getLogger().warn("Failed to decompress data", e); bos = null; } finally { CloseUtil.close(iis); CloseUtil.close(bis); CloseUtil.close(bos); } } return bos == null ? null : bos.toByteArray(); } /** * encode -- not implemented yet * */ public CachedData encode(Object o){ int flags = 0; byte[] b=null; return new CachedData(flags, b, getMaxSize()); } /** * no need to async Decode let's do it realtime */ public boolean asyncDecode(CachedData d) { return false; }}This is a stop-gap solution until we make the transition to Memcached with JSON encoding. Then I can use Jackson-JSON - which is a fast JSON encoder/decoder for Java enabling a great portable message protocol between the two stacks and nearly any other language we might add to the system (like Python).
-
Dot-Org Pavilion at the Percona Live MySQL Conference
Are you involved with an open-source project that’s interesting to MySQL users, such as Nginx, PHPMyAdmin, Drupal, Jenkins, PHP, and so on? Percona just published the application form for dot-org groups to have a free expo hall booth in the Percona Live MySQL Conference in April. Please submit your applications now, and tell your friends about this, because a) the schedule for applying is very short, and b) space is limited.
For those of you who don’t know what this is, it’s another of the O’Reilly traditions we’re trying to continue. (We are trying very hard to make this event as close to a clone of O’Reilly’s as we can.) It’s a free table in the expo hall where people who participate in a non-commercial open source project can exhibit. I organized a Maatkit booth a few times in the past, and was always really grateful to O’Reilly for making the space available. Space in the expo hall is at a premium, but we think that these dot-org booths are even more valuable to the open-source projects and the conference attendees.
So, please tell your friends who care about open source, and ask them to tell their friends too. Let’s get some great open-source projects into the expo hall, alongside the commercial vendors!
Further Reading:Percona Live MySQL Conference comes to Washington DC
One-day Percona Live conference in San Francisco
Maatkit at the dot-org pavilion at MySQL Conference and Expo 2009
Schedule for MySQL-and-beyond conference is live
Percona Live from a community insider’s perspective
-
MySQL progress in a year
Usually people do this around New Year, I will do it in February. Actually, I was inspired to do this after reviewing all the talks for this year's MySQL Conference - what a snapshot into the state of where we are! It made me realize we've made important progress in the past year, worth taking a moment to celebrate it. So here we go...
Diversification
In the past few years there was a lot of fear and doubt about MySQL due to Oracle taking over the ownership. But if you ask me, I was more worried for MySQL because of MySQL itself. I've often said that if MySQL had been a healthy open source project - like the other 3 components in the LAMP stack - then most of the NoSQL technologies we've seen come about would never have been started as their own projects, because it would have been more natural to build those needs on top of MySQL. You could have had a key-value store (HandlerSocket, Memcache API...), sharding (Spider) and clustering (Galera). You could have had a graph storage engine (OQGraph engine isn't quite it, I understand it is internally an InnoDB table still?). There could even have been MapReduce functionality, although I do think the Hadoop ecosystem targets problems that actually are better solved without MySQL.
read more
-
Upcoming talks at PHP UK, Confoo and your event
I will be speaking on new MySQL features at the PHP UK conference on February 25th in London. A few days later on march 3rd, I will be presenting the Care and Feeding of a MySQL Database at Confoo in Montreal. If you are attending either event, please introduce yourself!
Fellow Community Manager Keith Larson and I working hard to keep up with all the upcoming shows. If you think we might have not notices your upcoming event, please drop us a line. We will have thumb drive, baggage tags, stickers, and maybe (fingers crossed) t-shirts or plush dolphins soon.
-
New England’s Victory (for Big Data)
While it might not have been New England’s weekend on the Big Gridiron, it was certainly New England’s day for Big Data at the New England Database Summit on Friday at MIT.
The summit was well attended, with 350 registrants and keynotes from prominent MySQL users such as Mark Callaghan. The coverage was quite broad, with presentations running the gamut from grad students (complete with bodyguards and intimidating academic advisors) to established companies such as StreamBase. The sponsor list was an A-list this year as well, with EMC and Microsoft being the two biggest backers.
There were far too many and diverse topics to write about all of them. That said, here were a few of the notable ones.
Keynote #1: Johannes Gehrke (Cornell): Declarative Data-Driven Coordination
Johannes Gehrke of Cornell kicked off with the first keynote on Declarative-Driven Coordination. His methodology shed light on an alternative to out-of-band communication. The presentation focused on how to successfully handle entangled queries.
More Sleep for Tom and Meg if They Can Just Coordinate
In brief, what he showed is a way for someone to see if their friend is on a flight and have the database go about satisfying mutual constraints. With a proof that is outlined in his Sigmod paper, his main theorem is that any schedule that is entangled-isolated is also oracle-serializable. It’s a clever approach, as long as one’s set of friends being entangled remains small.
Keynote #2: Mark Callaghan (Facebook): Performance is Overrated
The room got a little quiet when Mark took the stage. Some people were expecting a possible rehash of this summer’s brouhaha between Mike Stonebraker and Facebook on the fate of MySQL. But, instead Mark jumped into some very practical discussions about managing MySQL at scale.
First, he noted that manageability needs more attention since…
The cost of extra hardware can be predicted
The cost of downtime cannot
Downtime comes in many forms (server down and server too busy)
For Mark, manageability has a number of meanings. This includes the rate of interrupts/server for the operations team. Mark finds that while the server count grows quickly, his operations team grows slowly. Hence, it is imperative that the quality-of-service improve over time (i.e., Does work get done? Does work get done on time?).
Mark and his team use MySQL for a number of reasons. First, it was there when Mark arrived. Second, Mark and his team made it scale 10x. Finally, Mark likes MySQL for OLTP.
As Facebook has grown though, so have the number of servers. This is due to “Big Data” x high QPS. Hence, they have had to add servers to add IOPs. To address this, Mark noted that flash memory (SSD) is very interesting as are (we blush) write-optimized databases.
The last part of his presentation focused on advice for scaling: More Data, More QPS. His tips were quite straightforward:
Fix stalls to make use of capacity
Don’t make MySQL faster, make it less slow
Improve efficiency to use less
Repeat
Additional details can be found in Sheeri’s excellent live blog of the presentation.
New Tools and Systems Session: Willis Lang (University of Wisconsin): Energy-Conscious Data Management Systems
Just as Mark stressed that performance isn’t everything when he spoke about management, Willis Lang pointed out another key concern. His slides noted that “three decades of database research has optimized for the highest possible performance possible regardless of energy consumption.” (We agree and have written about this topic as well).
Willis and his team have been looking at various techniques for addressing this such as using variable speed disks. He has been systematically studying the power/performance trade-offs of hardware components. The preliminary memory-based results showed that interesting trade-off opportunities exist if one rethinks database design principles. His presentation focused on the improvements that can be seen with memory parking. Additional details on his research can be found here.
As mentioned previously, there were many good talks — much more could be written about the event. Other interesting speakers included David Karger who introduced Dido, which seeks to make database manipulation as easy as document editing, and Alvin Cheung whose Pyxis project eases application development with automatic code partitioning based on application and server characteristics.
Kudos to Samuel Madden (MIT) and Ugur Cetintemel (Brown University) for organizing the event. Additional details can also be found via the Twitter hashtag #nedb12 and the event homepage.
|
News Feeds