tag:blogger.com,1999:blog-78038634470822007412024-03-14T10:06:31.893+01:00Paweł Hajdan's Dev BlogUnknownnoreply@blogger.comBlogger81125tag:blogger.com,1999:blog-7803863447082200741.post-39519498528124676522015-09-29T21:39:00.002+02:002015-09-29T21:39:41.008+02:00Gentoo's top-notch user communityOne of the best parts of Gentoo for me is the community.<br />
<br />
For example, I regularly receive email from people who volunteer to help with testing hard masked www-client/chromium ebuilds. FWIW you don't need to email me or the Gentoo Chromium team - just start testing and filing bugs. On the other hand, I do appreciate when people express interest in helping out.<br />
<br />
Another example is helping with getting bugs resolved.<br />
<br />
<a href="https://bugs.gentoo.org/show_bug.cgi?id=556812">Bug #556812</a> looked rather mysterious - until Guillaume ZITTA found that "build" is a red herring, and in fact the "tracing" module being imported is a different one (from dev-python/tracing as opposed to chromium sources). It was an unfortunate names collision - once found, quite easy to fix.<br />
<br />
In <a href="https://bugs.gentoo.org/show_bug.cgi?id=553502">bug #553502</a> Johan Hovold found we need to require postproc USE flag for libvpx to avoid a crash.<br />
<br />
See <a href="https://bugs.gentoo.org/show_bug.cgi?id=551666">bug #551666</a> for some Alexey Dobriyan's gdb magic mostly based on a single line segfault report from dmesg...<br />
<br />
These are just a few examples.<br />
<br />
By the way, the area where we could use more help is arm architecture support. Some specific bugs where help is wanted are <a href="https://bugs.gentoo.org/show_bug.cgi?id=555702">#555702</a>, <a href="https://bugs.gentoo.org/show_bug.cgi?id=545368">#545368</a>, <a href="https://bugs.gentoo.org/show_bug.cgi?id=520180">#520180</a>, and <a href="https://bugs.gentoo.org/show_bug.cgi?id=483130">#483130</a>. Please report whether you can reproduce or not, post emerge --info from your system and the compressed build log in case build fails.Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-7803863447082200741.post-76410955310671267722015-04-11T18:09:00.000+02:002015-04-11T18:09:35.135+02:00Tricks for resolving slot conflicts and blockersSlot conflicts can be annoying. It's worse when an attempt to fix them leads to an even bigger mess. I hope this post helps you with some cases - and that portage will keep getting smarter about resolving them automatically.<br>
<br>
<a href="http://phajdan-jr.blogspot.com/2015/04/tricks-for-resolving-slot-conflicts-and.html#more">Read more »</a>Unknownnoreply@blogger.com5tag:blogger.com,1999:blog-7803863447082200741.post-87510692045428726322015-03-05T19:58:00.000+01:002015-03-05T19:58:14.097+01:00More reliable handling of bash history across terminals and crashesI've been occasionally hitting frustrating issues with bash history getting lost after a crash. Then I found this great blog post about <a href="http://briancarper.net/blog/248/keeping-bash-history-in-sync-on-disk-and-between-multiple-terminals">keeping bash history in sync on disk and between multiple terminals</a>.<br />
<br />
tl;dr is to use "shopt -s histappend" and PROMPT_COMMAND="${PROMPT_COMMAND};history -a"<br />
<br />
The first is usually default, and results in sane behavior when you have multiple bash sessions at the same time. Now the second one ("history -a") is really useful to flush the history to disk in case of crashes.<br />
<br />
I'm happy to announce that both are now default in Gentoo! Please see <a href="https://bugs.gentoo.org/show_bug.cgi?id=517342">bug #517342</a> for reference.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-7803863447082200741.post-58145448704652764052014-08-07T09:20:00.000+02:002014-08-07T09:20:17.392+02:00Can your distro compile Chromium?Chromium is moving towards using C++11. Even more, it's going to require either <a href="https://groups.google.com/a/chromium.org/d/msg/chromium-dev/2RzwrIxnvqM/-o1V7fdTArcJ">gcc-4.8 or clang</a>.<br />
<br />
Distros like Ubuntu, Mageia, Fedora, openSUSE, Arch, CentOS, and Slackware are already using gcc-4.8 or later is their latest stable release.<br />
<br />
On the other hand, Debian Wheezy (7.0) has gcc-4.7.2. Gentoo is using gcc-4.7.3 in stable.<br />
<br />
I started a thread on gentoo-dev, <a href="http://thread.gmane.org/gmane.linux.gentoo.devel/92113">gcc-4.8 may be needed in stable for www-client/chromium-38.x</a>. There is a tracker for gcc-4.8 stabilization, <a href="https://bugs.gentoo.org/show_bug.cgi?id=516152">bug #516152</a>. There is also gcc-4.8 porting tracker, <a href="https://bugs.gentoo.org/show_bug.cgi?id=461954">bug #461954</a>.<br />
<br />
Please consider testing gcc-4.8 on your stable Gentoo system, and file bugs for any package that fails to compile or needs to have a newer version stabilized to work with new gcc. I have recompiled all packages, the kernel, and GRUB without problems.<br />
<br />
The title of this post is deliberately a bit similar to my earlier post <a href="http://phajdan-jr.blogspot.com/2010/09/is-your-distro-fast-enough-for-chromium.html">Is your distro fast enough for Chromium?</a> This browser project is pushing a lot towards shorter release cycles and latest software. I consider that a good thing. Now we just need to keep up with the updates, and any help is welcome.Unknownnoreply@blogger.com6tag:blogger.com,1999:blog-7803863447082200741.post-27017205714206343582014-07-19T19:19:00.001+02:002014-07-19T19:19:45.144+02:00Recovering from removed libgcc_s.so.1 (and missing busybox)<div class="p1">
I was experimenting in my arm chroot, and after a gcc upgrade and emerge --depclean --ask that removed the old gcc I got the following error:<br>
<span class="s1"><b><br></b></span>
<span style="font-family: Courier New, Courier, monospace;"><span class="s1"><b>#</b></span> ls -l</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace;">ls: error while loading shared libraries: libgcc_s.so.1: cannot open shared object file: No such file or directory</span><br>
<br>
Fortunately the newer working gcc was present, so the steps to make things work again were:</div>
<div class="p1">
<br></div>
<div class="p1">
</div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace;"><span class="s1"><b>#</b></span> LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/usr/lib/gcc/armv7a-hardfloat-linux-gnueabi/4.8.2/" gcc-config -l</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace;"> <span class="s2"><b>*</b></span> gcc-config: Active gcc profile is invalid!</span></div>
<div class="p2">
<span style="font-family: Courier New, Courier, monospace;"><br></span></div>
<div class="p1">
</div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace;"> [1] armv7a-hardfloat-linux-gnueabi-4.8.2</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace;"><br></span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace;"><span class="s1"><b>#</b></span> LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/usr/lib/gcc/armv7a-hardfloat-linux-gnueabi/4.8.2/" gcc-config 1 </span></div>
<div class="p1">
</div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace;"> <span class="s2"><b>*</b></span> Switching native-compiler to armv7a-hardfloat-linux-gnueabi-4.8.2 ...</span><br>
<br>
Actually my first thought was using busybox. The unexpected breakage during a routine gcc upgrade made me do some research in case I can't rely on /bin/busybox being present and working.<br>
<br>
I highly recommend the following links for further reading:<br>
<a href="http://lambdaops.com/rm-rf-remains">http://lambdaops.com/rm-rf-remains</a><br>
<a href="http://eusebeia.dyndns.org/bashcp">http://eusebeia.dyndns.org/bashcp</a><br>
<a href="http://www.reddit.com/r/linux/comments/27is0x/rm_rf_remains/ci199bk">http://www.reddit.com/r/linux/comments/27is0x/rm_rf_remains/ci199bk</a><br>
<br>
</div><a href="http://phajdan-jr.blogspot.com/2014/07/recovering-from-removed-libgccsso1-and.html#more">Read more »</a>Unknownnoreply@blogger.com3tag:blogger.com,1999:blog-7803863447082200741.post-30834199103423052962014-06-01T16:40:00.000+02:002014-06-01T16:40:20.447+02:00perl-cleaner slot conflict when upgrading perl (5.16 -> 5.18)If you tried upgrading from stable amd64 to ~amd64 or otherwise done a big update of perl, you probably hit this weird perl-cleaner slot conflict:<br>
<br>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"># perl-cleaner --all</span><br>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">!!! Multiple package instances within a single package slot have been pulled</span><br>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">!!! into the dependency graph, resulting in a slot conflict:</span><br>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br></span>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">dev-lang/perl:0</span><br>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br></span>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> (dev-lang/perl-5.18.2:0/5.18::gentoo, installed) pulled in by</span><br>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> =dev-lang/perl-5.18* required by (virtual/perl-IO-1.280.0:0/0::gentoo, ebuild scheduled for merge)</span><br>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> ^ ^^^^^ </span><br>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> dev-lang/perl:0/5.18=[-build(-)] required by (perl-core/version-0.990.800:0/0::gentoo, installed)</span><br>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> ^^^^^^^^ </span><br>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> (and 7 more with the same problems)</span><br>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br></span>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> (dev-lang/perl-5.16.3:0/5.16::gentoo, ebuild scheduled for merge) pulled in by</span><br>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> =dev-lang/perl-5.16* required by (virtual/perl-Package-Constants-0.20.0-r3:0/0::gentoo, installed)</span><br>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> ^ ^^^^^ </span><br>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> (and 6 more with the same problem)</span><br>
<div>
<br>
This is <a href="https://bugs.gentoo.org/show_bug.cgi?id=506616">bug #506616</a>, and the solution is to run the following command:<br>
<br></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">perl-cleaner --all -- --backtrack=30</span><br>
<br>
</div><a href="http://phajdan-jr.blogspot.com/2014/06/perl-cleaner-slot-conflict-when.html#more">Read more »</a>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-7803863447082200741.post-23592993730379974902014-01-20T23:32:00.000+01:002014-01-20T23:32:43.523+01:00System update report (last update a year ago)<div class="p1">
I have some systems that I update more rarely than the others. Of course I highly recommend keeping up to date, especially for the security fixes. I've also written similar update reports in the past. You may want to read them and compare the experience:<br>
<br>
<a href="http://phajdan-jr.blogspot.com/2011/10/another-5-month-update.html">Another 5-month update</a> (December 2011)<br>
<a href="http://phajdan-jr.blogspot.com/2012/09/another-report-from-rarely-updated.html">Another report from rarely updated system</a> (September 2012)<br>
<br>
</div><a href="http://phajdan-jr.blogspot.com/2014/01/system-update-report-last-update-year.html#more">Read more »</a>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-7803863447082200741.post-60624042205763679512013-12-04T05:58:00.003+01:002013-12-04T05:58:51.077+01:00Testers wanted: Aura in Chromium Dev channel (33.x)If you're using hard masked www-client/chromium dev channel packages (currently at version 33.x) you're probably used to testing things and encountering breakages from time to time.<br />
<br />
If you feel bored or adventurous, or both, or would just like to try the new Aura UI of Chromium (which will become default at some point), I encourage you to enable USE="aura" for www-client/chromium. Upstream still keeps it disabled by default, and that includes Google Chrome. In Gentoo it's easy to make a choice about this.<br />
<br />
Aura is a new UI architecture which is GPU accelerated. You can read about the technical details in its <a href="http://www.chromium.org/developers/design-documents/aura">documentation</a>. Especially because of the use of hardware acceleration it'd be useful to get many people to test it. If these changes break in your configuration, you can know now (while it's still early in development process), <a href="https://code.google.com/p/chromium/issues/list">report a bug</a> (consider posting the bug link in the comments so I can ensure it gets proper attention), and get back to the working configuration.<br />
<br />
Possible workaround for GPU problems is using --disable-gpu command-line flag. Note that this still means there is a bug that is unlikely to get fixed if you don't report it. It's also useful to include contents of about:gpu page in your bug report.<br />
<br />
Finally, see the screenshots below for an idea how it looks like (click thumbnails to see original images):<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-ps50saBr94o/Up60SZtLjTI/AAAAAAAAAPU/EQq02HEKmDk/s1600/chromium-aura.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="149" src="http://1.bp.blogspot.com/-ps50saBr94o/Up60SZtLjTI/AAAAAAAAAPU/EQq02HEKmDk/s200/chromium-aura.png" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">with aura</td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-_iIXNNOLDcU/Up60SgBqZNI/AAAAAAAAAPY/FbBm5wAATI0/s1600/chromium-no-aura.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="149" src="http://2.bp.blogspot.com/-_iIXNNOLDcU/Up60SgBqZNI/AAAAAAAAAPY/FbBm5wAATI0/s200/chromium-no-aura.png" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">without aura</td></tr>
</tbody></table>
<div style="clear: both;">
</div>
You can start compiling now, chromium-33.0.1726.0 has just been added to the tree.Unknownnoreply@blogger.com5tag:blogger.com,1999:blog-7803863447082200741.post-9111077428432279042013-11-26T01:11:00.002+01:002013-11-26T01:11:58.383+01:00Third party libraries in Chromium sourcesOne of the most common obstacles or issues related to inclusion of Chromium in Linux distro repositories are bundled libraries. Last attempt to blog about it I know about is Evan Martin's <a href="http://neugierig.org/software/chromium/notes/2009/12/forking.html">Forking upstream software</a> post. I decided to take another look.<br />
<div>
<br /></div>
<div>
It is important to know that even if something appears in a third_party directory in Chromium codebase, it is not necessarily a bundled library. Third party code - yes, but not necessarily a bundled library. What's the difference? Well, even Fedora in its excellent <a href="https://fedoraproject.org/wiki/Packaging:No_Bundled_Libraries">No Bundled Libraries</a> article lists e.g. copylibs as a possible exception. What about code that was never intended to be used as a shared library, is part of larger codebase, but is still useful? This will come up in some examples below.</div>
<div>
<br /></div>
<div>
Here is my list of third_party code still present in Gentoo's Chromium packages as of version 33.0.1711.3 (dev channel). This means that the libraries that have been successfully unbundled are not included. Similarly, code that is not used on Linux is not included. This takes into account intended audience - mostly Linux users, some of them packagers. A star (*) means that the code was already there in 2009.</div>
<div>
<ol>
<li>base/third_party/dmg_fp (*) - David M. Gay's floating point routines (dtoa, g_fmt). I don't think there is a shared library for these. There is also <a href="https://code.google.com/p/chromium/issues/detail?id=95729">crbug.com/95729</a> about using V8's routines.</li>
<li>base/third_party/dynamic_annotations - single .c file with corresponding header containing annotations for dynamic tools like valgrind, tsan. Doesn't seem to be worth extracting, which could likely add unwanted dependencies on these tools.</li>
<li>base/third_party/icu (*) - this could be extracted to use system icu; supporting nacl may be a challenge there (base is most likely also compiled by the nacl toolchain, and it's not obvious to me how shared libraries would work there - if at all, or whether it would make sense).</li>
<li>base/third_party/nspr (*) - it may be possible to remove it now that Gentoo dropped nacl support for other reasons (<a href="https://code.google.com/p/chromium/issues/detail?id=269560">crbug.com/269560</a>).</li>
<li>base/third_party/symbolize - part of another Google's project, <a href="https://code.google.com/p/google-glog/">google-glog</a>. Technically should be possible to extract, and glog even made a release with a tarball.</li>
<li>base/third_party/valgrind (*) - bundled to avoid depending on valgrind just for build... IMHO fine.</li>
<li>base/third_party/xdg_mime (*) - looks like the code was not intended to be used as a library, but maybe the intention was to avoid forking a process. Probably worth a closer look.</li>
<li>base/third_party/xdg_user_dirs (*) - see this comment in the source code:<br /><pre class="stx-plain" id="code" style="background-color: white; line-height: 16.25px; padding-top: 0.5em; position: relative; z-index: 50;"><span class="stx-line" id="code_1" style="display: block;"><span class="stx-comment" style="color: #880000;">/*</span>
</span><span class="stx-line" id="code_2" style="display: block;"><span class="stx-comment" style="color: #880000;"> This file is not licenced under the GPL like the rest of the code.</span>
</span><span class="stx-line" id="code_3" style="display: block;"><span class="stx-comment" style="color: #880000;"> Its is under the MIT license, to encourage reuse by cut-and-paste.</span>
</span><span class="stx-line" id="code_4" style="display: block;">
</span><span class="stx-line" id="code_5" style="display: block;"><span class="stx-comment" style="color: #880000;"> Copyright (c) 2007 Red Hat, inc</span></span><span class="stx-line" id="code_5" style="display: block;"><span class="stx-comment" style="color: #880000;"> ...</span></span><span class="stx-line" id="code_5" style="display: block;"><span style="color: #880000;">*/
</span></span></pre>
</li>
<li>breakpad/src/third_party/curl - great candidate for unbundling (or just disabling breakpad for Chromium builds in a way that doesn't try to touch curl even when disabled).</li>
<li>chrome/third_party/mozilla_security_manager - parts of Mozilla code; doesn't seem to be designed as a shared library; has local modifications.</li>
<li>crypto/third_party/nss - selected files extracted from NSS; there are some modifications, but with enough effort it may be possible to unbundle.</li>
<li>net/third_party/mozilla_security_manager - parts of Mozilla code, different from the chrome bits above.</li>
<li>net/third_party/nss (*) - parts of Mozilla's NSS (libssl) with experimental patches. Note that NSS developer is working there, so this can be seen as even more bleeding-edge than NSS trunk.</li>
<li>third_party/WebKit (*) - now Blink, developed as part of the same Chromium project, but a fork of third party code. Not designed to be used as a shared library.</li>
<li>third_party/angle_dx11 - developed by Google/Chromium developers; doesn't seem to be designed to be used as a shared library, but with enough effort it should be possible.</li>
<li>third_party/cacheinvalidation - same as above.</li>
<li>third_party/cld (*) - developed by Google/Chromium developers, probably to be replaced with cld2, which will hopefully be closer to a shared library design.</li>
<li>third_party/cros_system_api - related to ChromeOS, not really bundled but rather just part of the project.</li>
<li>third_party/ffmpeg (*) - Chrome uses very recent ffmpeg; I think the local modifications status has improved greatly since 2009: looks like patches make it upstream pretty quickly.</li>
<li>third_party/flot - JS library, AFAIK there isn't really a concept of having system JS libraries. It could actually be useful to have one, but it's not obvious.</li>
<li>third_party/hunspell (*) - modified to support running under sandbox and loading dictionaries in a different format; maintainers do respond but are very busy. This is doable but requires a fair amount of effort to figure out what to do with the different dictionary format.</li>
<li>third_party/iccjpeg - taken out of lcms library, and the maintainers <a href="http://sourceforge.net/mailarchive/message.php?msg_id=30221395">don't want to expose it</a>.</li>
<li>third_party/jstemplate (*) - Google's JS templating library.</li>
<li>third_party/khronos - GL headers, unfortunately with local modifications.</li>
<li>third_party/leveldatabase - needs a redesign to allow applying Chromium-specific behavior (env_chromium.cc) at run-time instead of at compile time. I've seen a Debian package for leveldb, looks like there is some interest in using it as a library.</li>
<li>third_party/libjingle (*) - used to have semi-inactive upstream, now seems to become a part of WebRTC. When things stabilize more, worth another look.</li>
<li>third_party/libphonenumber - upstream seems to be more focused on Java version of it, which actually has releases; C++ version doesn't seem to be designed to be used as a shared library.</li>
<li>third_party/libsrtp - used to be inactive but now has a new home at <a href="https://github.com/cisco/libsrtp">https://github.com/cisco/libsrtp</a> and there are Googlers helping out with it. Worth taking another look when things stabilize. Note that even if it compiles it doesn't mean it works, see <a href="https://bugs.gentoo.org/show_bug.cgi?id=459932">bug #459932</a>.</li>
<li>third_party/libusb - locally made incompatible change needs to be upstreamed (<a href="https://code.google.com/p/chromium/issues/detail?id=266149">crbug.com/266149</a>).</li>
<li>third_party/libvpx - waiting for upstream release supporting vp9, see <a href="https://bugs.gentoo.org/show_bug.cgi?id=487926">bug #487926</a>.</li>
<li>third_party/libwebp - waiting for upstream release supporting APIs Chromium depends on, see <a href="http://crbug.com/288019">http://crbug.com/288019</a>.</li>
<li>third_party/libxml/chromium - this is ugly: code is actually part of Chromium codebase; at least it's not really bundled.</li>
<li>third_party/libXNVCtrl - part of nvidia-settings. Not sure if it's intended to be used as a shared library, but it seems totally possible technically, and I even remember some success reports with it.</li>
<li>third_party/libyuv - Google/Chromium project. Should be possible to use as a shared library, but doesn't seem to make releases.</li>
<li>third_party/lss - Linux Syscall Support; a header based on Linux kernel headers.</li>
<li>third_party/lzma_sdk (*) - lzma library from 7-zip.org ; it would be great to <a href="https://groups.google.com/a/chromium.org/d/msg/chromium-packagers/LEUnzNzzRTQ/xz_SCnZakskJ">replace it with xz-utils</a> which distros package.</li>
<li>third_party/mesa - I think only headers are used, but it's <a href="https://groups.google.com/a/chromium.org/d/msg/chromium-packagers/amyyjkDj7Cg/cl5up-ABW7oJ">complicated</a>.</li>
<li>third_party/modp_b64 (*) - README.chromium points to <a href="https://code.google.com/p/stringencoders/">https://code.google.com/p/stringencoders/</a>. Doesn't seem to be design to be used as a shared library, but it seems possible.</li>
<li>third_party/mt19937ar - not designed as a shared library, rather small; can be removed after move to C++11 (looks like <random> would support needed functionality).</li>
<li>third_party/npapi (*) - NPAPI headers with modifications.</li>
<li>third_party/ots (*) - OpenType sanitizer, may be possible to package as a shared library, although it doesn't seem to have releases.</li>
<li>third_party/polymer - JS library by Google, see <a href="http://www.polymer-project.org/">polymer-project.org</a>.</li>
<li>third_party/pywebsocket (*) - Python WebSocket server used for testing. Should be possible to package it separately.</li>
<li>third_party/qcms - color management library. Last upstream commits seem to be over a year ago, but the bundled copy continued to receive various updates, at least for more recent toolchain support.</li>
<li>third_party/sfntly - font-related library; doesn't seem to have releases, doesn't seem to be designed to be a shared library.</li>
<li>third_party/skia (*) - graphics library, changes very often.</li>
<li>third_party/smhasher - hash function library - doesn't seem to have releases or be designed to be a shared library.</li>
<li>third_party/sqlite (*) - available as a package, the biggest obstacle is lack of a good API to use it in a multi-process sandboxed context and also test it. See <a href="http://crbug.com/22208">http://crbug.com/22208</a>. That obstacle would disappear however when Chromium drops support for abandoned <a href="http://www.w3.org/TR/webdatabase/">webdatabase spec</a>.</li>
<li>third_party/tcmalloc (*) - although theoretically available separately, the Chromium copy is heavily modified, and that includes hardening changes important for security.</li>
<li>third_party/tlslite (*) - Python crypto library, only used for testing but appears to be modified in a non-compatible way.</li>
<li>third_party/trace-viewer - not obvious what it really is, and it contains several more bundled libraries inside.</li>
<li>third_party/undoview - code extracted from gtksourceview.</li>
<li>third_party/usrsctp - user-space SCTP implementation with local changes.</li>
<li>third_party/webdriver - mostly some minified JS embedded in C++ code.</li>
<li>third_party/webrtc - Real-Time Communications library - doesn't seem to have releases, and seems to be moving pretty fast.</li>
<li>third_party/widevine - stubs for proprietary content distribution module.</li>
<li>third_party/x86inc - asm code extracted from x264 with local modifications; I don't really see a good way to provide that as a system package.</li>
<li>third_party/zlib/google - this is ugly: code is actually part of Chromium codebase; at least it's not really bundled.</li>
<li>url/third_party/mozilla - parts of Mozilla code; doesn't seem to be designed as a shared library; has local modifications.</li>
<li>v8 (*) - although the path doesn't contain third_party, I consider it bundled code. See <a href="http://phajdan-jr.blogspot.com/2013/11/when-libraries-you-use-are-moving-too.html">When the libraries you use are moving too fast</a> for the reasons it's there. While technically not part of Evan's 2009 list, it was obviously there since the beginning.</li>
</ol>
<div>
60 entries look like a lot. I would like that number to be smaller. On the other hand, note that many of these codebases were not designed to be used as shared libraries, some were developed as part of Chromium project, and that the project is very careful to put code it borrows from outside in third_party directories, whereas it's not uncommon for open source projects in general to incorporate such code directly into their codebases. In Chromium it's just much more visible.</div>
</div>
<div>
<br /></div>
<div>
Also note that while 23 of these items still exist, for some entries from 2009 we're now using system libraries, at least in Gentoo. Just to give you a few examples (the list is not necessarily complete - star means it's present on the 2009 list):</div>
<div>
<ol>
<li>flac</li>
<li>harfbuzz (*)</li>
<li>icu (*)</li>
<li>jsoncpp</li>
<li>libevent (*)</li>
<li>libjpeg (*)</li>
<li>libpng (*)</li>
<li>libxml (*)</li>
<li>libxslt (*)</li>
<li>minizip</li>
<li>nspr (*)</li>
<li>openssl</li>
<li>opus</li>
<li>protobuf (*)</li>
<li>re2</li>
<li>snappy</li>
<li>speex</li>
<li>xdg-utils (*)</li>
<li>yasm (*)</li>
<li>zlib (*)</li>
</ol>
<div>
I'm interested in your opinions, so feel free to add your comment below. If you liked this post, you may also like <a href="http://phajdan-jr.blogspot.com/2013/01/state-of-chromium-open-source-packages.html">State of Chromium Open Source packages</a>.</div>
</div>
Unknownnoreply@blogger.com3tag:blogger.com,1999:blog-7803863447082200741.post-48803001609326432852013-11-18T02:05:00.000+01:002013-11-18T02:08:16.606+01:00When the libraries you use are moving too fastRecently I masked dev-lang/v8 on Gentoo with the following message:<br /><br />
<pre style="white-space: pre-wrap; word-wrap: break-word;"># Pawel Hajdan jr (13 Nov 2013)
# Masked for removal in 30 days. Does not have stable
# API resulting in compile breakages in reverse
# dependencies. Combined with short release cycle
# (6 weeks) this makes it pretty much unusable as
# a shared library. See <a href="https://bugs.gentoo.org/417879">bug #417879</a>, <a href="https://bugs.gentoo.org/420995">bug #420995</a>,
# <a href="https://bugs.gentoo.org/471582">bug #471582</a>, <a href="https://bugs.gentoo.org/477300">bug #477300</a>, <a href="https://bugs.gentoo.org/484786">bug #484786</a>, <a href="https://bugs.gentoo.org/490214">bug #490214</a>.
# Also, the following discussions:
# - <a href="http://thread.gmane.org/gmane.linux.gentoo.devel/88222">http://thread.gmane.org/gmane.linux.gentoo.devel/88222</a><u>
</u># - <a href="http://thread.gmane.org/gmane.linux.gentoo.devel/88811">http://thread.gmane.org/gmane.linux.gentoo.devel/88811</a><u>
</u>dev-lang/v8
</pre>
All packages depending on shared v8 library are now either bundling it or are masked.<br />
<br />
This is obviously a quite controversial change. People are opposed to bundling libraries for good reasons. I'd like to make it clear that I'm also strongly in favor of using system libraries when possible. I'm also pragmatic though: in case of v8 this resulted in multiple bug experienced by users - just see the links above. With the API of v8 changing every 6 weeks and security fixes being pushed every now and then, these other packages depending on v8 just don't keep up.<br />
<br />
Now the v8 team has made some nice improvements, as you can see on <a href="https://code.google.com/p/v8/wiki/Source">https://code.google.com/p/v8/wiki/Source</a>:<br />
<blockquote class="tr_bq">
<span style="background-color: white; font-family: arial, sans-serif; font-size: 13px; line-height: 16.399999618530273px;">V8 public API (basically the files under include/ directory) may change over time. New types/methods may be added without breaking existing functionality. When we decide that want to drop some existing class/methods, we first mark it with </span><a href="https://code.google.com/p/chromium/codesearch#search/&q=V8_DEPRECATED&sq=package:chromium&type=cs" rel="nofollow" style="background-color: white; color: #0000cc; font-family: arial, sans-serif; font-size: 13px; line-height: 16.399999618530273px;">V8_DEPRECATED</a><span style="background-color: white; font-family: arial, sans-serif; font-size: 13px; line-height: 16.399999618530273px;"> macro which will cause compile time warnings when the deprecated methods are called by the embedder. We keep deprecated method for one branch and then remove it. E.g. if </span><tt style="background-color: white; font-family: Monaco, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console', monospace; font-size: 12px; line-height: 16.399999618530273px; max-width: 66em;">v8::CpuProfiler::FindCpuProfile</tt><span style="background-color: white; font-family: arial, sans-serif; font-size: 13px; line-height: 16.399999618530273px;"> was plain non deprecated in </span><i style="background-color: white; font-family: arial, sans-serif; font-size: 13px; line-height: 16.399999618530273px;">3.17</i><span style="background-color: white; font-family: arial, sans-serif; font-size: 13px; line-height: 16.399999618530273px;"> branch, marked as </span><tt style="background-color: white; font-family: Monaco, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console', monospace; font-size: 12px; line-height: 16.399999618530273px; max-width: 66em;">V8_DEPRECATED</tt><span style="background-color: white; font-family: arial, sans-serif; font-size: 13px; line-height: 16.399999618530273px;"> in </span><i style="background-color: white; font-family: arial, sans-serif; font-size: 13px; line-height: 16.399999618530273px;">3.18</i><span style="background-color: white; font-family: arial, sans-serif; font-size: 13px; line-height: 16.399999618530273px;">, it may well be removed in </span><i style="background-color: white; font-family: arial, sans-serif; font-size: 13px; line-height: 16.399999618530273px;">3.19</i><span style="background-color: white; font-family: arial, sans-serif; font-size: 13px; line-height: 16.399999618530273px;"> branch.</span></blockquote>
Indeed I see V8_DEPRECATED being used in new v8 changes instead of removing APIs immediately. I heard sometimes they need to remove things immediately anyway, when some APIs are inherently buggy and lead to memory errors like leaks or double-frees.<br />
<br />
Ideally I'd like to see a longer grace period for removal of APIs (not just one release, since it's only 6 weeks). Then maybe ABI stability in the scope of one release could become a consideration. The consistent usage of V8_DEPRECATED will for sure lead to more data about how much more effort is now needed to maintain the V8 API. If it turns out to be manageable, I hope to see these further improvements being tried.<br />
<br />
By the way, with ffmpeg/libav we're pretty much in a very similar situation: Chromium uses bleeding-edge ffmpeg code, and other packages just can't keep up with API updates. Sometimes the APIs Chromium depends on are not part of any ffmpeg/libav release: Chromium developers actively contribute to upstream ffmpeg codebase and it's reasonable to iterate quickly instead of waiting for a release. I think even Fedora has exceptions for bundling libraries in such circumstances.<br />
<br />
I don't expect this post to appease most people from the packaging community. This is just stating where we currently are. Maybe having some more stable layer that could be added on top of Chromium codebase in a manner similar to <a href="https://github.com/01org/ozone-wayland">Ozone-Wayland</a> would be one way. Still, that'd be a considerable engineering effort, mostly to keep old things working rather than developing the future. I think it has some value, the main question is just what to do with limited time developers have.<br />
<br />
<i>I'm experimentally enabling comments for this post - feel free to share your thoughts.</i>Unknownnoreply@blogger.com8tag:blogger.com,1999:blog-7803863447082200741.post-37407743706057972682013-04-15T21:25:00.000+02:002013-04-15T21:25:10.466+02:00Best articles about Blink rendering engine according to meIt is now over a week since <a href="http://blog.chromium.org/2013/04/blink-rendering-engine-for-chromium.html">announcement of Blink</a>, a rendering engine for the Chromium project.<br />
<br />
I hope it could be useful to provide links to the best articles about it, which have good, technical contents.<br />
<br />
<a href="http://blog.html5test.com/post/47128015347/thoughts-on-blink">Thoughts on Blink</a> from HTML5 Test is a good summary about history of Chrome, WebKit, and puts this recent announcement in context. For even more context (nothing about Blink) you can read Paul Irish's excellent <a href="http://paulirish.com/2013/webkit-for-developers/">WebKit for Developers</a> post.<br />
<br />
Peter-Paul Koch (probably best known for quirksmode.org) has good articles about Blink: <a href="http://www.quirksmode.org/blog/archives/2013/04/blink.html">Blink</a> and <a href="http://www.quirksmode.org/blog/archives/2013/04/blinkbait.html">Blinkbait</a>.<br />
<br />
I also found it interesting to ready Krzysztof Kowalczyk's <a href="https://medium.com/my-ideas/25a947158087">Thoughts on Blink</a>.<br />
<br />
Highly recommended Google+ posts by Chromium developers:<br />
<ul>
<li>Justin Schuh talks about <a href="https://plus.google.com/u/0/116560594978217291380/posts/AeCnq76cAXb">security improvements</a> made possible by Blink</li>
<li>Ben Goodger explains <a href="https://plus.google.com/u/0/105636695715347097518/posts/Ubrgmz3LpaR">layering and embedding</a> of the new engine</li>
<li>Charlie Reis is excited about <a href="https://plus.google.com/u/0/105497998876878526147/posts/etnTiaXZEGM">out-of-process iframes</a></li>
</ul>
<div>
If you're interested in the technical details or want to participate in the discussions, why not follow <a href="https://groups.google.com/a/chromium.org/forum/?fromgroups#!forum/blink-dev">blink-dev</a>, the mailing list of the project?</div>
Unknownnoreply@blogger.comtag:blogger.com,1999:blog-7803863447082200741.post-51303801452103691192013-02-09T22:38:00.000+01:002013-02-09T22:38:47.738+01:00Bumpy upgrades: udev-171 -> udev-197, iptables-1.4.13 -> iptables-1.4.16.3I guess many people may hit similar problems, so here is my experience of the upgrades. Generally it was pretty smooth, but required paying attention to the details and some documentation/forums lookups.<br />
<br />
<b>udev-171 -> udev-197 upgrade</b><br />
<br />
<ol>
<li>Make sure you have CONFIG_DEVTMPFS=y in kernel .config, otherwise the system becomes unbootable for sure (I think the error message during boot mentions that config option, which is good).</li>
<li>The ebuild also asks for CONFIG_BLK_DEV_BSG=y, not sure if that's strictly needed but I'm including it here for completeness.</li>
<li>Things work fine for me <i>without</i> DEVTMPFS_MOUNT. I haven't tried with it enabled, I guess it's optional.</li>
<li>I do not have a split /usr. YMMV then if you do.</li>
<li>Make sure to run "rc-update del udev-postmount".</li>
<li><b>Expect network device names to change</b> (I guess this is a non-issue for systems with a single network card). This can really mess up things in quite surprising ways. It seems /etc/udev/rules.d/70-persistent-net.rules no longer works (<a href="https://bugs.gentoo.org/show_bug.cgi?id=453494">bug #453494</a>). Note that the "new way" to do the same thing (<a href="http://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames">http://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames</a>) is disabled by default in Gentoo (see /etc/udev/rules.d/80-net-name-slot.rules). For now I've adjusted my firewall and other configs, but I think I'll need to figure out the new persistent net naming system.</li>
</ol>
<br />
<b>iptables-1.4.13 -> iptables-1.4.16.3</b><br />
<b><br /></b>
<span style="font-family: Courier New, Courier, monospace;"> * Loading iptables state and starting firewall ... <br />WARNING: The state match is obsolete. Use conntrack instead. <br />iptables-restore v1.4.16.3: state: option "--state" must be specified</span><div>
<br /></div>
<div>
It can be really non-obvious what to do with this one. Change your rules from e.g. "-m state --state RELATED" to "-m conntrack --ctstate RELATED". See <a href="http://forums.gentoo.org/viewtopic-t-940302.html">http://forums.gentoo.org/viewtopic-t-940302.html</a> for more info.<br /> Also note that iptables-restore doesn't really provide good error messages, e.g. "iptables-restore: line 48 failed". I didn't find a way to make it say what exactly was wrong (the line in question was just a COMMIT line, it didn't actually identify the real offending line). These mysterious errors are usually caused by missing kernel support for some firewall features/targets.<br />
<br />
<b>two upgrades together</b><br />
<br />
Actually what adds to the confusion is having these two upgrades done simultaneously. This makes it harder to identify which upgrade is responsible for which breakage. For an even smoother ride, I'd recommend upgrading iptables first, making sure the updated rules work, and then proceed with udev.</div>
Unknownnoreply@blogger.comtag:blogger.com,1999:blog-7803863447082200741.post-17588944189392285182013-01-28T15:56:00.000+01:002013-01-28T15:56:21.345+01:00State of Chromium Open Source packagesLet me present an informal an unofficial state of Chromium Open Source packages as I see it. Note a possible bias: I'm a Chromium developer (and this post represents my views, not the projects'), and a Gentoo Linux developer (and Chromium package maintenance lead - this is a team effort, and the entire team deserves credit, especially for keeping stable and beta ebuilds up to date).<br />
<br />
<ol>
<li><b>Gentoo Linux</b> - ships stable, beta and dev channels. Security updates are promptly pushed to stable. NaCl (NativeClient) is enabled, although pNaCl (Portable NaCl) is disabled. Up to 23 use_system_... gyp switches are enabled (depending on USE flags).</li>
<li><b>Arch Linux</b> - ships stable channel, promptly reacts to security updates. NaCl is enabled, following Gentoo closely - I consider that good, and I'm glad people find that code useful. :) 5 use_system_... gyp switches are enabled. A notable thing is that the PKGBUILD is one of the shortest and simplest among Chromium packages - this seems to follow from <a href="https://wiki.archlinux.org/index.php/The_Arch_Way">The Arch Way</a>. There is also <a href="https://aur.archlinux.org/packages/chromium-dev/">chromium-dev on AUR</a> - it is more heavily based on the Gentoo package, and tracks the upstream dev channel. Uses 19 use_system_... gyp switches.</li>
<li><b>FreeBSD / OpenBSD</b> - ship stable channel, and are doing pretty well, especially when taking amount of BSD-specific patching into account. NaCl is disabled.</li>
<li><b>ALT Linux</b> - ships stable channel. NaCl seems to be disabled by default, I'm not sure what's actually shipped in compiled package. Uses 11 use_system_... gyp switches.</li>
<li><b>Debian</b> - ancient 6.x version in Squeeze, 22.x in sid at the time of this writing. This is two major milestones behind, and is missing security updates. <i>Not recommended at this moment.</i> :( If you are on Debian, my advice is to use Google Chrome, since official debs should work, and monitor state of the open source Chromium package. You can always return to it when it gets updated.</li>
<li><b>Fedora</b> - not in official repositories, but Tom "spot" Callaway has an unofficial repo. Note: currently the version in that repo is 23.x, one major version behind on stable. Tom wrote an article in 2009 called <a href="http://spot.livejournal.com/312320.html">Chromium: Why it isn't in Fedora yet as a proper package</a>, so there is definitely an interest to get it packaged for Fedora, which I appreciate. Many of the issues he wrote about are now fixed, and I hope to work on getting the remaining ones fixed. Please stay tuned!</li>
</ol>
<div>
This is not intended to be an exhaustive list. I'm aware of openSUSE packages, there seems to be something happening for Ubuntu, and I've heard of Slackware, Pardus, PCLinuxOS and CentOS packaging. I do not follow these closely enough though to provide a meaningful "review".</div>
<div>
<br /></div>
<div>
Some conclusions: different distros package Chromium differently. Pay attention to the packaging lag: with about 6 weeks upstream release cycle and each major update being a security one, this matters. Support for NativeClient is another point. There are extension and Web Store apps that use it, and when more and more sites start to use it, this will become increasingly important. Then it is interesting why on some distros some bundled libraries are used even though upstream provides an option to use a system library that is known to work on other distros.</div>
<div>
<br /></div>
<div>
Finally, I like how different maintainers look at each other's packages, and how patches and bugs are frequently being sent upstream.</div>
Unknownnoreply@blogger.comtag:blogger.com,1999:blog-7803863447082200741.post-91693132884053373752013-01-04T21:23:00.000+01:002013-01-04T21:23:31.288+01:00Signal handler safety, re-entering mallocThis is a story from real-world development. From signal(7):<br />
<br />
<br />
<span style="font-family: Courier New, Courier, monospace;"> <b>Async-signal-safe functions</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"> A signal handler function must be very careful,</span><br />
<span style="font-family: Courier New, Courier, monospace;"> since processing elsewhere may be interrupted at some</span><br />
<span style="font-family: Courier New, Courier, monospace;"> arbitrary point in the </span><span style="font-family: 'Courier New', Courier, monospace;">execution of the program.</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> POSIX has the concept of "safe function". If a signal</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> interrupts the execution of an unsafe func</span><span style="font-family: 'Courier New', Courier, monospace;">tion,</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> and handler calls an unsafe function, then the behavior</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> of the program is undefined.</span><br />
<div>
<br /></div>
<div>
After that a list of safe functions follows, and one notable things is that <b>malloc and free are async-signal-unsafe!</b></div>
<div>
<b><br /></b></div>
<div>
I hit this issue while enabling tcmalloc's debugallocation for Chromium Debug builds. We have a StackDumpSignalHandler for tests, which prints a stack trace on various crashing signals for easier debugging. It's very useful, and worked fine for a pretty long while (which means that "but it works!" is not a valid argument for doing unsafe things).</div>
<div>
<br /></div>
<div>
Now when I enabled debugallocation, I noticed hangs triggered by the stack trace display. In one example, this stack trace:</div>
<div>
<br /></div>
<div>
<pre style="background-color: white; font-size: 12px; max-width: 80em; padding-left: 0.7em; white-space: pre-wrap;">@0 0x00000000019c6c85 in tcmalloc::Abort () at third_party/tcmalloc/chromium/src/base/abort.cc:15
@1 0x00000000019b39c1 in LogPrintf (severity=-4,
pat=0x32aeb18 "memory allocation/deallocation mismatch at %p: allocated with %s being deallocated with %s", ap=0x7fff52c379e8)
at third_party/tcmalloc/chromium/src/base/logging.h:210
@2 0x00000000019b3a8b in RAW_LOG (lvl=-4,
pat=0x32aeb18 "memory allocation/deallocation mismatch at %p: allocated with %s being deallocated with %s")
at third_party/tcmalloc/chromium/src/base/logging.h:230
@3 0x00000000019c3fb1 in MallocBlock::CheckLocked (this=0x7fd18f143400, type=-21308287)
at ./third_party/tcmalloc/chromium/src/debugallocation.cc:461
@4 0x00000000019c3c42 in MallocBlock::CheckAndClear (this=0x7fd18f143400, type=-21308287)
at ./third_party/tcmalloc/chromium/src/debugallocation.cc:401
@5 0x00000000019c436a in MallocBlock::Deallocate (this=0x7fd18f143400, type=-21308287)
at ./third_party/tcmalloc/chromium/src/debugallocation.cc:557
@6 0x00000000019c1929 in DebugDeallocate (ptr=0x7fd18f143420, type=-21308287)
at ./third_party/tcmalloc/chromium/src/debugallocation.cc:998
<b>@7 0x00000000028d1482 in tc_delete (p=0x7fd18f143420) at ./third_party/tcmalloc/chromium/src/debugallocation.cc:1232</b>
@8 0x000000000097dc04 in cc::ResourceProvider::deleteResourceInternal (this=0x7fd191827da0, it=...) at cc/resource_provider.cc:242
@9 0x000000000097daaf in cc::ResourceProvider::deleteResource (this=0x7fd191827da0, id=1) at cc/resource_provider.cc:230
@10 0x00000000006f9824 in (anonymous namespace)::ResourceProviderTest_Basic_Test::TestBody (this=0x7fd18dc5abf0)
at cc/resource_provider_unittest.cc:328
@11 0x00000000008ec801 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x7fd18dc5abf0,
method=&virtual testing::Test::TestBody(), location=0x29463ab "the test body") at testing/gtest/src/gtest.cc:2071
@12 0x00000000008e9665 in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x7fd18dc5abf0,
method=&virtual testing::Test::TestBody(), location=0x29463ab "the test body") at testing/gtest/src/gtest.cc:2123
@13 0x00000000008dee0d in testing::Test::Run (this=0x7fd18dc5abf0) at testing/gtest/src/gtest.cc:2143
@14 0x00000000008df3ea in testing::TestInfo::Run (this=0x7fd191823020) at testing/gtest/src/gtest.cc:2319
@15 0x00000000008df8dc in testing::TestCase::Run (this=0x7fd19181f0d0) at testing/gtest/src/gtest.cc:2426
@16 0x00000000008e3eea in testing::internal::UnitTestImpl::RunAllTests (this=0x7fd19829dd60) at testing/gtest/src/gtest.cc:4249</pre>
<pre style="background-color: white; font-size: 12px; max-width: 80em; padding-left: 0.7em; white-space: pre-wrap;">
</pre>
generates SIGSEGV (tcmalloc::Abort). This is just debugallocation having stricter checks about usage of dynamically allocated memory. Now the StackDumpSignalHandler kicks in, and internally calls malloc. But we're already inside malloc code as you can see on the above stack trace (see frame @7, bold font), and re-entering it tries to take locks that are already held, resulting in a hang.</div>
<div>
<br /></div>
<div>
<a href="https://chromiumcodereview.appspot.com/11362048">The fix</a> required several changes:</div>
<div>
<ul>
<li>no dynamic memory, and that includes std::string and std::vector, which use it internally</li>
<li>no buffered stdio or iostreams, they are not async-signal-safe (that includes fflush)</li>
<li>custom code for number-to-string conversion that doesn't need dynamically allocated memory (snprintf is not on the list of safe functions as of POSIX.1-2008; it seems to work on a glibc-2.15-based system, but as said before this is not a good assumption to make); in this code I've named it itoa_r, and it supports both base-10 and base-16 conversions, and also negative numbers for base-10</li>
<li>warming up backtrace(3): now this is really tricky, and backtrace(3) itself is not whitelisted for being safe; in fact, on the very first call it does some memory allocations; for now I've just added a call to backtrace() from a context that is safe and happens before the signal handler may be executed; implementing backtrace(3) in a known-safe way would be another fun thing to do</li>
</ul>
</div>
<div>
Note that for the above, I've also added a unit test that triggers the deadlock scenario. This will hopefully catch cases where calling backtrace(3) leads to trouble.</div>
<div>
<br /></div>
<div>
For more info, feel free to read the articles below:</div>
<div>
<ul>
<li><a href="https://www.securecoding.cert.org/confluence/display/seccode/SIG30-C.+Call+only+asynchronous-safe+functions+within+signal+handlers">CERT Secure Coding: Call only asynchronous-safe functions within signal handlers</a></li>
<li><a href="http://lcamtuf.coredump.cx/signals.txt">Michał Zalewski (lcamtuf): Delivering Signals for Fun and Profit</a></li>
</ul>
</div>
Unknownnoreply@blogger.comtag:blogger.com,1999:blog-7803863447082200741.post-20158000259850415242012-11-30T01:25:00.002+01:002012-11-30T01:25:34.198+01:00Google Chrome / Chromium : Failed to move to new PID namespace: Cannot allocate memoryIf you're seeing a message like "Failed to move to new PID namespace: Cannot allocate memory" when running Chrome, this is actually a problem with the Linux kernel.<br />
<br />
For more context, see <a href="http://code.google.com/p/chromium/issues/detail?id=110756">http://code.google.com/p/chromium/issues/detail?id=110756</a> . In case you wonder what the fix is, the patch is available at <a href="http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commitdiff;h=976a702ac9eeacea09e588456ab165dc06f9ee83">http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commitdiff;h=976a702ac9eeacea09e588456ab165dc06f9ee83</a>, and it should be in Linux-3.7-rc6.Unknownnoreply@blogger.comtag:blogger.com,1999:blog-7803863447082200741.post-47509747820541269172012-09-28T10:52:00.000+02:002012-09-28T10:52:33.100+02:00Debugging SELinux file context mismatchesI originally posted the question on gentoo-hardened ML, but Sven Vermeulen advised me to file a bug, so there it is: <a href="https://bugs.gentoo.org/show_bug.cgi?id=436474">bug #436474</a>.<br />
<br />
The problem I hit is that my <span style="white-space: pre-wrap;">~/.config/chromium/ directory should have </span><span style="white-space: pre-wrap;">unconfined_u:object_r:chromium_xdg_config_t context, but it has </span><span style="white-space: pre-wrap;">unconfined_u:object_r:xdg_config_home_t instead.</span><br />
<span style="white-space: pre-wrap;"><br /></span>
<span style="white-space: pre-wrap;">I could manually force the "right" context, but it turned out even removing the directory in question and allowing the browser to re-create it still results in wrong context. Looks like something deeper is broken (maybe just on my system), and fixing the root cause is always better. After all, other people may hit this problem too.</span><br />
<span style="white-space: pre-wrap;"><br /></span>
<span style="white-space: pre-wrap;">Here is what error messages appear on chromium launch:</span><br />
<span style="white-space: pre-wrap;"><br /></span>
<br />
<pre class="bz_comment_text" id="comment_text_0" style="white-space: pre-wrap; width: 50em;">$ chromium
[2557:2557:1727940797:ERROR:process_singleton_linux.cc(263)] Failed to
create /home/ph/.config/chromium/SingletonLock: Permission denied
[2557:2557:1727941544:ERROR:chrome_browser_main.cc(1552)] Failed to
create a ProcessSingleton for your profile directory. This means that
running multiple instances would start multiple browser processes rather
than opening a new window in the existing process. Aborting now to avoid
profile corruption.</pre>
<pre class="bz_comment_text" id="comment_text_0" style="white-space: pre-wrap; width: 50em;">
</pre>
And SELinux messages:<pre class="bz_comment_text" id="comment_text_0" style="white-space: pre-wrap; width: 50em;">
</pre>
<pre class="bz_comment_text" id="comment_text_0" style="white-space: pre-wrap; width: 50em;"># audit2allow -d
#============= chromium_t ==============
allow chromium_t xdg_config_home_t:file create;
allow chromium_t xdg_config_home_t:lnk_file { read create };
[ 107.872466] type=1400 audit(1348505952.982:67): avc: denied { read
} for pid=2166 comm="chrome" name="SingletonLock" dev="sda1" ino=522327
scontext=unconfined_u:unconfined_r:chromium_t
tcontext=unconfined_u:object_r:xdg_config_home_t tclass=lnk_file
[ 107.873916] type=1400 audit(1348505952.983:68): avc: denied {
create } for pid=2178 comm="Chrome_FileThre"
name=".org.chromium.Chromium.ZO3dGF"
scontext=unconfined_u:unconfined_r:chromium_t
tcontext=unconfined_u:object_r:xdg_config_home_t tclass=file</pre>
<pre class="bz_comment_text" id="comment_text_0" style="white-space: pre-wrap; width: 50em;">
</pre>
If you have any ideas how to further debug it, or how to solve it, please share (e.g. comment on the bug or send me an e-mail). Thanks!Unknownnoreply@blogger.comtag:blogger.com,1999:blog-7803863447082200741.post-65934257993590538512012-09-20T17:35:00.001+02:002012-09-20T17:35:40.334+02:00Stabilization hiccup with dev-perl/net-server-2.6.0What happened?<br />
<br />
<b>Sep 13th</b> I stabilized net-analyzer/munin-2.0.5-r1 (<a href="https://bugs.gentoo.org/show_bug.cgi?id=412881">security bug #412881</a>). I use automated repoman checks and USE="-ipv6", and everything was fine at the time I committed the stabilization (also, see no mention of net-server in that security bug).<br />
<br />
<b>Sep 14th</b> Seraphim Mellos filed <a href="https://bugs.gentoo.org/show_bug.cgi?id=434978">bug #434978</a> about munin pulling in ~arch net-server.<br />
<br />
<b>Sep 16th</b> x86@ team has been re-added to <a href="https://bugs.gentoo.org/show_bug.cgi?id=412881">security bug #412881</a>. Meanwhile Mr_Bones_ pinged me on irc. Also, Diego Elio Pettenò (flameeyes) filed <a href="https://bugs.gentoo.org/show_bug.cgi?id=435242">bug #435242</a> against repoman not catching the dependency problem.<br />
<br />
<b>Sep 17th</b> I stabilized dev-perl/net-server-2.6.0 on x86, fixing the immediate problem.<br />
<br />
<b>Sep 18th</b> the repoman fix has been released in portage-<span style="white-space: pre-wrap;">2.1.11.18 and 2.2.0_alpha129.</span><br />
<span style="white-space: pre-wrap;"><br /></span>
<span style="white-space: pre-wrap;">Now the only remaining thing to do is pushing the portage/repoman fix to stable. I especially like how quickly the fix for root cause (repoman check) has been produced and released.</span>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-7803863447082200741.post-27804405873728181382012-09-04T13:05:00.000+02:002012-09-04T13:05:39.320+02:00Another report from rarely updated systemThis is another (second) post about updating a system I rarely updated. If you're interested, read the <a href="http://phajdan-jr.blogspot.com/2011/10/another-5-month-update.html">first post</a>. I recommend more frequent updates, but I also want to show that it's possible to update without re-installing, and how to solve common problems.<br>
<br>
<a href="http://phajdan-jr.blogspot.com/2012/09/another-report-from-rarely-updated.html#more">Read more »</a>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-7803863447082200741.post-53456688251054800722012-05-19T19:38:00.001+02:002012-05-19T19:38:17.525+02:00revdep-rebuild doesn't detect qt-core's dependency on libicui18n.so.48It's a known issue, <a href="https://bugs.gentoo.org/show_bug.cgi?id=413541">bug #413541</a>. The end result is weird, because revdep-rebuild tells you that everything is fine, yet some apps display errors (they still launch though):<br />
<br />
<br />
<pre class="bz_comment_text" id="comment_text_0" style="white-space: pre-wrap; width: 50em;"><span style="font-family: 'Courier New', Courier, monospace;">Unable to load library icui18n "Cannot load library icui18n: (libicui18n.so.48: cannot open shared object file: No such file or directory)"</span></pre>
<pre class="bz_comment_text" id="comment_text_0" style="white-space: pre-wrap; width: 50em;">
</pre>
The workaround is to re-emerge qt-core (more packages might be affected).<br />
<br />
If you wonder what's the root cause, it's using dlopen (in this case in qt-core) instead of linking directly (e.g. ELF DT_NEEDED entry) with given library.<br />
<br />
In binary-only world, using dlopen may make sense sometimes. Package names, versions, and library SONAMEs vary between distros, so it's difficult to create a single package that works with multiple distros. It may be easier to dlopen the needed libraries (sometimes even trying different SONAMEs), and fail gracefully in case they are missing (e.g. just disable some optional functionality).<br />
<br />
That's what Qt is doing here. However, in Open Source world, where software is packaged by distributions, the above case should be handled by linking directly (DT_NEEDED), allowing tools like revdep-rebuild to detect the breakage.<br />
<br />
Other distributions are hitting this problem too, see e.g. <a href="https://bugzilla.redhat.com/show_bug.cgi?id=759923">https://bugzilla.redhat.com/show_bug.cgi?id=759923</a> and <a href="https://bugs.launchpad.net/ubuntu/+source/qt4-x11/+bug/989915">https://bugs.launchpad.net/ubuntu/+source/qt4-x11/+bug/989915</a>.Unknownnoreply@blogger.comtag:blogger.com,1999:blog-7803863447082200741.post-48499146802892932562012-05-12T13:47:00.002+02:002012-09-10T09:17:51.074+02:00ffmpeg saves the day (.mts files)If you need to convert .mts files to .mov (so that e.g. iMovie can import them), I found ffmpeg to be the best tool for the task (I don't want to install and run "free format converters" that are usually Windows-only and come from untrusted sources). This post is inspired by <a href="http://blog.laaz.org/tech/2010/03/18/imovie-and-mts/">iMovie and MTS</a> blog post.<br />
<br />
First I tried just changing the container:<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">for x in *.MTS; do ffmpeg -i ${x} -c copy ${x/.MTS/.mov}; done</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span>
<br />
But QuickTime could not play sound from those files because of <a href="http://en.wikipedia.org/wiki/Dolby_AC-3">AC-3 codec</a>. Also, the quality of the video playback was very poor. The other command I tried was:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">for x in *.MTS; do ffmpeg -i ${x} -vcodec copy -acodec mp2 -ac 2 ${x/.MTS/.mov}; done</span><br />
<br />
Now QuickTime was able to play the sound, but problems with video remained. iMovie was unable to import the resulting files anyway (silently: I got no error message, just nothing happened when trying to import).<br />
<br />
<b>The final command that is proven to work well is this:</b><br />
<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">for x in *.MTS; do ffmpeg -i ${x} -vcodec mpeg1video -acodec mp2 -ac 2 -sameq ${x/.MTS/.mov}; done</span><br />
<br />
The video has been converted perfectly, and iMovie successfully imported the movies. Note the useful bash substitution of extension, ${x/.MTS/.mov}. Enjoy!<br />
<br />
<br />
<br />
<br />Unknownnoreply@blogger.comtag:blogger.com,1999:blog-7803863447082200741.post-82930283849302023992012-05-04T16:52:00.002+02:002012-05-14T14:05:17.435+02:00Another reason why SELinux's neverallow is very usefulI'm only beginning my experiments with SELinux, and neverallow (which is basically an assertion that prevents inserting certain allow rules) seemed a bit weird to me.<br />
<br />
While experimenting with some local policies though, after an update (selinux-base-policy and other sec-policy packages) my local policy triggered a neverallow rule about sys_module capability being unnecessarily granted.<br />
<br />
In fact, re-compiling the local policy and loading the new version made the error disappear. Now this is indeed useful, because binary policy files are arguably harder to inspect, and if they get out of sync with the base policy, it's easy to introduce errors like in this case.<br />
<br />
Another conclusion is that learning takes time: it was the update that triggered this situation. I wonder what else awaits me in the SELinux land. ;-)Unknownnoreply@blogger.comtag:blogger.com,1999:blog-7803863447082200741.post-57424051002272162732012-04-21T22:44:00.000+02:002012-04-23T14:57:01.118+02:00Watch out for shadow-4.1.5-r1 and pambase-20120417I recently bumped shadow-4.1.5-r1 and pambase-20120417. <a href="https://bugs.gentoo.org/show_bug.cgi?id=412721">bug #412721</a> has been filed about people being locked out after the update.<br />
<br />
The fix is to run dispatch-conf. The reports are unclear, but you might not get any message from emerge that it is necessary. If dispatch-conf doesn't update anything, make sure to manually re-emerge pambase.<br />
<br />
What's going on? /etc/pam.d/{login,passwd,su} are being moved from shadow package to pambase, so they can be shared with hardened-shadow. They are not really shadow-specific, but Gentoo-specific, so it makes sense to make pambase own those files.Unknownnoreply@blogger.comtag:blogger.com,1999:blog-7803863447082200741.post-24607145373737241932012-04-16T08:23:00.000+02:002012-04-16T08:23:12.832+02:00www-client/chromium and libjpeg-turboRecently <a href="https://bugs.gentoo.org/show_bug.cgi?id=412031">Gentoo bug #412031</a> (">=www-client/chromium-18.0 should not force the media-libs/libjpeg-turbo dependency") has been filed. Samuli did the right thing and closed it, but I'd like to use this as an opportunity to explain the issue in more detail.<br />
<br />
The original bug is <a href="https://bugs.gentoo.org/show_bug.cgi?id=393471">#393471</a>, filed by Gentoo Chromium team member Mike Gilbert four months ago and fixed in February. An <a href="http://trac.webkit.org/changeset/101286">optimization</a> done in WebKit JPEG decoding code (which speeds up JPEG decoding by up to 2x) assumed that the code will run against the same version of JPEG library it has been compiled with. In general such assumptions do not hold (with the note of SONAMEs, i.e. same library SONAME should mean it's binary-compatible; more on that later), but Google Chrome bundles all of those third-party libraries anyway, so for Google it's true.<br />
<br />
I started a <a href="https://groups.google.com/a/chromium.org/group/chromium-dev/browse_thread/thread/571bc73228177c63?pli=1">discussion</a> on chromium-dev mailing list, and also had direct e-mail exchange with developers doing the WebKit and libjpeg-turbo changes. The end result is we'd either have to keep a local patch in Gentoo to revert the optimization, or just always depend on libjpeg-turbo.<br />
<br />
Of course the most cool would be to do the optimization The Right Way, by doing runtime detection of JPEG extensions, as made possible by DRC, libjpeg-turbo developer and explained in <a href="https://bugs.gentoo.org/show_bug.cgi?id=393471#c23">bug #393471 comment 23</a>. jcstest.c in libjpeg-turbo source tree indeed shows how to do that. The only reason I haven't done that yet in WebKit is lack of time (it's not just producing the patch; compiling WebKit and running its layout tests takes a lot of it; and then dealing with any possible breakages), but I plan to do that in the future (help is welcome).<br />
<br />
Speaking about The Right Way, another broken thing is that libraries that have the same SONAME (in this case libjpeg and libjpeg-turbo) behave differently when using JPEG extensions. You may recognize the libpng "breakages", but it seems to me libpng upstream actively cleans up the interface of the library, and it obviously updates SONAME accordingly. That's a good thing.<br />
<br />
Hopefully it's now more clear why the Gentoo package www-client/chromium depends on libjpeg-turbo and not virtual/jpeg.Unknownnoreply@blogger.comtag:blogger.com,1999:blog-7803863447082200741.post-48468612424820335362012-04-10T20:58:00.000+02:002012-05-14T14:05:17.434+02:00Experimenting with SELinuxRecently I started experimenting with SELinux and created a new VM for that. I've used the excellent <a href="http://www.gentoo.org/proj/en/hardened/selinux/selinux-handbook.xml">Gentoo SELinux Handbook</a> and mostly got things running. I also submitted <a href="https://bugs.gentoo.org/show_bug.cgi?id=411005">bug #411005</a>, <a href="https://bugs.gentoo.org/show_bug.cgi?id=411365">bug #411365</a>, and <a href="https://bugs.gentoo.org/show_bug.cgi?id=411377">bug #411377</a> to make it even better.<br />
<br />
SELinux may seem scary or complicated at the first glance, but my experience so far was pretty good. I did encounter some AVC denials, but that was fine. Here are my general notes about running a SELinux system:<br />
<br />
<ol>
<li><b>UNIX permissions work normally. </b>If UNIX permissions deny access, SELinux rules are not checked. Access is just denied, as expected. SELinux rules can result in more denials, but can never allow more than UNIX permissions.<br /></li>
<li><b>Permissive mode is really useful.</b> In that mode the denials are only logged, not enforced. The system should work as usual, and you have chance to note and fix eventual problems. You can actually switch between permissive and enforcing on the fly (setenforce 1, setenforce 0), so you can experiment whether things would work in enforcing mode and quickly switch back otherwise.<br /></li>
<li><b>Targeted policy is recommended for desktop systems. </b>It's part of general advice to "start small", but desktop systems have many more available applications anyway. Usually servers are minimal installations, so it's easier to write complete strict policy for them. Targeted SELinux policy uses <i>unconfined_t</i> context by default, which has no restrictions at all. Only selected applications have a specific policy just for them.<br /></li>
<li><b>You don't need to worry about SELinux users with targeted policy.</b> Another part that makes SELinux more complicated is that normal Linux users are mapped to SELinux users. This is important when using strict policy, but with the targeted one everyone is just mapped to <i>unconfined_u</i>.<br /></li>
<li><b>Most access decisions are made based just on the type.</b> Usual SELinux context, e.g. <i>unconfined_u:object_r:user_home_t</i> may seem quite complicated. However, corresponding allow rule would most likely only need to mention the type, i.e. <i>user_home_t </i>(to be precise, two types: types of the object and type of the subject).<br /></li>
<li><b>Many confusions are just matter of vocabulary.</b> For example, there is no practical difference between domain and type (<a href="http://billauer.co.il/selinux-policy-module-howto.html">source</a>). My understanding is that domains are types used to confine processes, so every domain is a type, but not every type is a domain.<br /></li>
<li><b>Allow rules are actually pretty simple.</b> For example <i>"allow example_t self:process execmem;".</i> Ignore the syntax, the meaning of each rule is really straightforward, and I think it's no different with alternative MAC (Mandatory Access Control) implementations. And when writing a policy for specific domain (in this case <i>example_t</i>), all of the rules are going to start with <i>"allow example_t ...".</i></li>
</ol>
<div>
For further reading, I recommend <a href="http://www.imperialviolet.org/2009/07/14/selinux.html">SELinux from the inside out</a> and <a href="http://billauer.co.il/selinux-policy-module-howto.html">Breaking the Ice with SELinux</a>.</div>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-7803863447082200741.post-35145600891529644192012-02-23T11:23:00.000+01:002012-02-23T11:23:59.236+01:00Some bugs detected during arch testingJust wanted to share some list of bugs I've recently hit during arch testing / stabilization. Especially useful for that is <a href="http://phajdan-jr.blogspot.com/2011/10/exhaustive-testing-of-stable-reverse.html">testing of reverse dependencies</a>.<br />
<br />
<a href="https://bugs.gentoo.org/show_bug.cgi?id=387789">bug #387789</a> - app-office/passepartout-0.6_p1 failed (compile phase)<br />
<a href="https://bugs.gentoo.org/show_bug.cgi?id=390805">bug #390805</a> - dev-embedded/ftdi_eeprom-0.2 fails to compile<br />
<a href="https://bugs.gentoo.org/show_bug.cgi?id=397513">bug #397513</a> - dev-util/netbeans-6.8-r1 fails to compile (JavacTask.java:107: package CheckForCleanBuilds does not exist)<br />
<a href="https://bugs.gentoo.org/show_bug.cgi?id=398881">bug #398881</a> - media-libs/libxspf-1.2.0 fails to install (/usr/bin/install: cannot stat `html/*.gif': No such file or directory)<br />
<a href="https://bugs.gentoo.org/show_bug.cgi?id=398887">bug #398887</a> - dev-db/pgadmin3-1.14.1 pre-merge check failed (postgresql-config: command not found)<br />
<a href="https://bugs.gentoo.org/show_bug.cgi?id=399469">bug #399469</a> - dev-python/matplotlib-1.0.1-r1 fails to build with USE=doc<br />
<a href="https://bugs.gentoo.org/show_bug.cgi?id=403851">bug #403851</a> - dev-java/xmlunit-1.3 uses network during src_compile<br />
<a href="https://bugs.gentoo.org/show_bug.cgi?id=403859">bug #403859</a> - dev-db/slony1-1.2.10 fails to compile (error: 'SerializableSnapshot' undeclared)<br />
<br />
And bugs that are fixed now:<br />
<br />
<a href="https://bugs.gentoo.org/show_bug.cgi?id=385403">bug #385403</a> - dev-php5/pecl-http-{1.7.0-r1,1.7.1} sandbox violation<br />
<a href="https://bugs.gentoo.org/show_bug.cgi?id=385423">bug #385423</a> - games-simulation/crashtest-1.1 fails to build (fltk-config misuse?)<br />
<a href="https://bugs.gentoo.org/show_bug.cgi?id=387785">bug #387785</a> - dev-cpp/gtksourceviewmm-2.10.1 failed (glibmm missing shared doc utilities?)<br />
<a href="https://bugs.gentoo.org/show_bug.cgi?id=390831">bug #390831</a> - media-sound/amarok-2.4.3 fails to compile (config-amarok.h: No such file or directory)<br />
<a href="https://bugs.gentoo.org/show_bug.cgi?id=394995">bug #394995</a> - emerge: InvalidDependString: USE flag 'birdstep' is not in IUSE (REQUIRED_USE contains flag not in IUSE)<br />
<a href="https://bugs.gentoo.org/show_bug.cgi?id=399513">bug #399513</a> - dev-lang/xsb-3.3.2 fails to compile<br />
<a href="https://bugs.gentoo.org/show_bug.cgi?id=399621">bug #399621</a> - media-plugins/vdr-dvd-0.3.7_pre20071113-r1 fails to compile against media-libs/libdvdnav-4.2.0<br />
<a href="https://bugs.gentoo.org/show_bug.cgi?id=401137">bug #401137</a> - dev-java/mx4j-tools-3.0.2 fails to unpack with jythonUnknownnoreply@blogger.com