The core of the core of the big data solutions -- Map
Title: The core of the core of the big data solutions -- Map
Platform: Windows, linux
Technology: Perfect hash algorithm
Description: Map algorithm with high performance
Section MFC c++ map stl
SubSection c++ algorithm
For the c++ program, map is used everywhere.And bottleneck of program perfo=
rmance is often the performance of map.Especially in the case of large data=
,and the business association closely and unable to realize the data distri=
bution and parallel processing condition.So the performance of map becomes =
the key technology.
In the work experience with telecommunications industry and the information=
security industry, I was dealing with the big bottom data,especially the m=
ost complex information security industry data,all can't do without map.
For example, IP table, MAC table, telephone number list, domain name resolu=
tion table, ID number table query, the Trojan horse virus characteristic co=
de of cloud killing etc..
The map of STL library using binary chop, its has the worst performance.Goo=
gle Hash map has the optimal performance and memory at present, but it has =
repeated collision probability.Now the big data rarely use a collision prob=
ability map,especially relating to fees, can't be wrong.
Now I put my algorithms out here,there are three kinds of map,after the bui=
ld is Hash map.We can test the comparison,my algorithm has the zero probabi=
lity of collision,but its performance is also better than the hash algorith=
m, even its ordinary performance has no much difference with Google.
My algorithm is perfect hash algorithm,its key index and the principle of c=
ompression algorithm is out of the ordinary,the most important is a complet=
ely different structure,so the key index compression is fundamentally diff=
erent.The most direct benefit for program is that for the original map need=
ten servers for solutions but now I only need one server.
Declare: the code can not be used for commercial purposes, if for commercia=
l applications,you can contact me with QQ 75293192.
First,modern warfare can't be without the mass of information query, if the=
query of enemy target information slows down a second, it could lead to th=
e delaying fighter, leading to failure of the entire war. Information retri=
eval is inseparable from the map, if military products use pwwhashMap inste=
ad of the traditional map,you must be the winner.
Scond,the performance of the router determines the surfing speed, just repl=
ace open source router code map for pwwHashMap, its speed can increase ten =
There are many tables to query and set in the router DHCP ptotocol,such as =
IP,Mac ,and all these are completed by map.But until now,all map are using=
STL liabrary,its performance is very low,and using the Hash map has error =
probability,so it can only use multi router packet dispersion treatment.If =
using pwwHashMap, you can save at least ten sets of equipment.
Third,Hadoop is recognized as the big data solutions at present,and its mos=
t fundamental thing is super heavy use of the map,instead of SQL and table.=
Hadoop assumes the huge amounts of data so that the data is completely unab=
le to move, people must carry on the data analysis in the local.But as long=
as the open source Hadoop code of the map changes into pwwHashMap, the per=
formance will increase hundredfold without any problems.
Background to this article that may be useful such as an introduction to th=
e basic ideas presented: