No, not yet?

Ok, well, in that case, here comes rescue: With the little script described here, it becomes really easy to start Infinite Scale to check it out on your computer. It makes it really easy and quick for you, without Linux super admin powers whatsoever.

To use it, you just need to open a terminal on your machine and cd into a directory somewhere in your home where you can afford to host some bytes.

Without further preparation, you type the following command line (NOT as user root please):

curl -L https://owncloud.com/runocis.sh | /bin/bash

What it does is that it automatically pulls the latest stable version of Infinite Scale from the official download server of ownCloud onto your computer. For that, it creates a configuration and a start script, and starts the server. The script detects the platform on which your’re running to download the right binary version. It also looks up the hostname and configures the installation for that name.

Once the server was started Infinite Scale’s web client can be accessed by pointing a browser to the URL https://your-hostname:9200/. Since this is an installation for testing purposes, it does not have a proper certificate configured. That is why your browser is complaining about the cert, and you have to calm it. And indeed, that is one of the reasons why you’re not supposed to use this sneak peak in production or even exposed to the internet.

For the nerds, the script does not really do magic, but just curls the golang single binary of Infinite Scale down to the machine into a sandbox directory, chmod it to be executable and create a working config and a data dir. All happens with the priviledges of the logged in user, no sudo or root involved. You’re encouraged to double check the install script using for example the command curl -L https://owncloud.com/runocis.sh | less - of course you never should trust anybody running scripts from the internet on your machine.

If the server is stopped by pressing Ctrl-C, it later can be started again by the script runocis.sh that was kindly left behind in the sandbox as well.

The installer was tested on these three platforms: 64 bit AMD/Intel CPU based Linux machines, 64 bit Raspberry Pi with Raspbian OS and MacOSX. The flavour of Linux should not make a difference.

If you encounter a problem with the script or if you have suggestions to improve, please find it in my this’n that section on Github. I am happy to receive issue reports or pull requests.

For further information and setups suitable for production please refer to the Infinite Scale documentation.

One of the topics is the VFS (Virtual Filesystem-) API for the Linux desktop. Indeed that is a topic for us at ownCloud as well, and I like to share our perspective on it, discussing it in the scope of the free desktop.

The topic is very important, as “syncing” of data from and to cloud storages has changed over time. From having all files mirrored from client to server and vice versa, it has now shifted to keep all files in the cloud, and have them as so called placeholders on the desktop. That means that most files on the client appear with size zero to save space, but the complete filesystem structure is available.

If a user starts to interact with such a dehydrated file, the content is of the file is downloaded transparently utilizing the cloud system client, for example ownClouds desktop client. The same happens when an application accesses such a file. As a result, the placeholders look and behave like the normal filesystem we are used to.

On Windows and on MacOSX, the problem is kind of solved. Both have added APIs to their OS that can be used to implement the access of data on the cloud.

On Linux, we do not have this kind of API yet. That means that it is close to impossible to implement this user experience. Volker already said that desktop environment specific solutions probably do not scale, which I agree with.

At ownCloud we have looked into the implementation of a specific FUSE file system. That should certainly be possible, and is probably a part of the solution, but is considerable effort because of the asynchronous nature of the topic. Given that the market share of Linux desktop systems is pretty small it is not attractive for companies to invest a lot into a Linux only system. Here the power of community could make a difference again.

It would be best if we as open source community would come up with a shared solution as a free desktop standard, that might be oriented on one of the existing APIs, maybe the MacOSX File Provider API: A library and little framework that the linux desktop environments can work with abstracting the VFS.

While collaborating on that, all data clouds could implement the bindings to their storage. With that, the extra implementation efforts for the Linux solution hopefully wouldnt be dramatic any more.

Let’s call this system openVFS as a work title. How can we evolve it? I’d like to invite all interested parties to discuss in this temporary Github repo to collect ideas and opinions. There is also a little experimental code.

Fine with me. Just to support that, I did a little hackery over XMas to improve the support for ownClouds virtual file system on the Linux Desktop.

What are Virtual Files?

In professional usecases, users often have a huge amount of data stored in ownCloud. Syncing these completely to the desktop computer or laptop would be too much and costly in bandwidth and harddisk space. That is why most mature file sync solutions came up with the concept of virtual files. That means that users have the full structure with directories and files mirrored to their local machines, but have placeholder of the real files in the local file manager.
The files, however, are not on the disk. They get downloaded on demand.

That way, users see the full set of data virtually, but save time and space of files they never will need on the local system.

The ownCloud Experience

ownCloud has innovated on that topic a while ago, and meanwhile support virtual files mainly on Windows, because there is an elaborated system API to work with placeholder files. As we do not have this kind of API on Linux desktops yet, ownCloud desktop developers implemented the following solution for Linux: The virtual files are 1 byte sized files with the name of the original file, plus a suffix “.owncloud” to indicate that they are virtual.

That works, yet has one downside: Most file managers do not display these placeholder files nicely, because they loose the MIME type information. Also, downloading and also freeing up the space of downloaded files is not integrated. To summarize, there is a building block missing to make that useful on Linux.

Elokab-files-manager with ownCloud

I was wondering if it wasn’t possible to change an existing file manager to support the idea of virtual files. To start trying I found the Elokab-files-manager which is a very compact, yet feature rich file manager built on Qt. It is built with very few other dependencies. Perfect to start playing around with.

In my Github fork you can see the patches I came up with to make Elokab-fm understand ownCloud virtual files.

New Functionality

The screenshot shows the changes in the icon view of Elokab-fm.

Screenshot of patched Elokab-files-manager to support ownCloud Virtual Files.

To make that possible, Elokab-fm now pulls some information from the ownCloud Sync Client config file and connects to the sync client via local socket to share some information. That means, that the sync client needs to run to make that work.

Directories that are synced with ownCloud now show an cloud overlay in the center (1).

The placeholder files (2) which are not present on the local hard drive indicate that by showing a little cloud icon bottom right. However, other than before, they are displayed with their correct name and mime-type, which makes this already much more useful.

Files, which are on the local disk as the image (3) show their thumbnail as usual.

In the side panel (4) there are a few details added: The blue box on the bottom indicates that the file manager is connected to the sync client. For the selected virtual file (2), it shows an button that downloads the file if clicked which would turn it into a non virtual, local file. There is also an entry in the context menu to achieve that.

Status

This is just a proof of concept and a little XMas fun project. There are bugs, and the implementation is not complete. And maybe Jürgen’s idea of a FUSE layer to improve that is the better approach, but anyway. It just shows what is possible with virtual files also on Linux.

If you like that idea, please let us know or send a little PR if you like. We do not wanna fail providing our share on the year of the Linux Desktop, right? ;-)

Building it from my Github branch should be fairly easy, as it only depends on Qt.

For openSUSE users, I will provide some test packages in my home project on Open Build Service.

Some people think it is a great idea to include the functionality in the sync client.

I don’t agree because it combines two very complex topics into one code base and makes the code difficult to maintain. The risk is high to end up with a kind of code base which nobody is able to maintain properly any more. So let’s better avoid that for ownCloud and look for alternatives.

A good way is to use a so called encrypted overlay filesystem and let ownCloud sync the encrypted files. The downside is that you can not use the encrypted files in the web interface because it can not decrypt the files easily. To me, that is not overly important because I want to sync files between different clients, which probably is the most common usecase.

Encrypted overlay filesystems put the encrypted data in one directory called the cipher directory. A decrypted representation of the data is mounted to a different directory, in which the user works.

That is easy to setup and use, and also in principle good to use with file sync software like ownCloud because it does not store the files in one huge container file that needs to be synced if one bit changes as other solutions do.

To use it, the cypher directory must be configured as local sync dir of the client. If a file is changed in the mounted dir, the overlay file system changes the crypto files in the cypher dir. These are synced by the ownCloud client.

One of the solutions I tried is CryFS. It works nicely in general, but is unfortunately very slow together with ownCloud sync.

The reason for that is that CryFS is chunking all files in the cypher dir into 16 kB blocks, which are spread over a set of directories. It is very beneficial because file names and sizes are not reconstructable in the cypher dir, but it hits on one of the weak sides of the ownCloud sync. ownCloud is traditionally a bit slow with many small files spread over many directories. That shows dramatically in a test with CryFS: Adding eleven new files with a overall size of around 45 MB to a CryFS filesystem directory makes the ownCloud client upload for 6:30 minutes.

Adding another four files with a total size of a bit over 1MB results in an upload of 130 files and directories, with an overall size of 1.1 MB.

A typical change use case like changing an existing office text document locally is not that bad. CryFS splits a 8,2 kB big LibreOffice text doc into three 16 kB files in three directories here. When one word gets inserted, CryFS needs to create three new dirs in the cypher dir and uploads four new 16 kB blocks.

My personal conclusion: CryFS is an interesting project. It has a nice integration in the KDE desktop with Plasma Vault. Splitting files into equal sized blocks is good because it does not allow to guess data based on names and sizes. However, for syncing with ownCloud, it is not the best partner.

If there is a way how to improve the situation, I would be eager to learn. Maybe the size of the blocks can be expanded, or the number of directories limited? Also the upcoming ownCloud sync client version 2.6.0 again has optimizations in the discovery and propagation of changes, I am sure that improves the situation.

Let’s see what other alternatives can be found.

[caption id=“attachment_943” align=“alignright” width=“338”] Code lines changed as of Nov. 10, 2018[/caption]

That motivated me to do some investigation what the Nextcloud client actually consists of (at due date Nov. 10, 2018). I was looking into the NC desktop client git repoository grouped the numbers of commits of people that can be associated clearly to either the ownCloud- or Nextcloud project, or to “other communities” or machine commits. Since the number of commits could be misleading (maybe some commits are huge?) I did the same exercise with numbers of changed lines of code.

When looking on the changed lines, the first top six contributors to the Nextcloud desktop client are only active in the ownCloud project. Number seven is an “other community” contributor whos project the client was based on in the beginning. Number eight to eleven go to Nextcloud, with a low percentage figure.

[caption id=“attachment_944” align=“alignnone” width=“666”] # of commits to the Nextcloud Desktop repository as of Nov. 10, 2018[/caption]

As a result, far more than 80% of the changed lines of the Nextcloud client is actually work that ownClouders did (not considering the machine commits). In the past, and also today. The number would be even higher if it considered all the commits that go into the NC repo with an NC author, but are actually ownCloud patches where the original author got lost on the way by merging them through a NC branch. It looks like the Nextcloud developers were actually adding less commits to their client than all “other community” developers so far.

No wonder, it is a fork, you might think, and that is of course true. However, to my taste these numbers are not reflecting a “constructive” fork driving things forward when we talk about sync technology.

That is all fine, and I am proud that the work we do in ownCloud is actually stimulating two projects, with different focus areas nowadays. On the other hand, I would appreciate if the users of the technology would take a closer look to understand who really innovates, drives things forward and also fixes the nasty bugs in the stack. As a matter of fairness, that should be acknowledged. That is the motivation that keeps free software contributors busy and communities proud.

Recently an ownCloud fork was coming up with a little shiny box with one harddisk, that can be complemented with a Rapsberry Pi and their software, promoting that as your private cloud.

While I like the idea of building a private cloud for everybody (I started to work on ownCloud because of that idea back in the days), I do not think that this example of gear is a good solution for private cloud.

In fact I believe that throwing this kind of implementations on the table is especially unfortunate because if we come up with too many not optimal proposals, we waste the  willingness of users to try it. This idea should not target geeks who might be willing to try ideas on and on. The idea of the private cloud needs to target at every computer user who wants to store data safely, but does not want to care about longer than ever necessary. And with them I fear we only have very little chances, if one at all, to introduce them to a private cloud solution before they go back to something that simply works.

Here are some points why I think solutions like the proposed one are not good enough:

Hardware

That is nothing new: The hardware of the Raspberry Pi was not designed for this kind of usecases. It is simply too weak to drive ownCloud, which is an PHP app plus database server that has some requirements on the servers power. Even with PHP7, which is faster, and the latest revisions of the mini computer, it might look ok in the beginning, but after all the neccessary bells and whistles were added to the installation and data run in, it will turn out that the CPU power is simply not enough. Similar weaknesses are also true for the networking capabilities for example.

A user that finds that out after a couple of weeks after she worked with the system will remain angry and probably go (back) to solutions that we do not fancy.

One Disk Setup

The solution comes as one disk setup: How secure can data be that is on one single hardisk? A seriously engineered solution should at least recommend a way to store the data more securely and/or backup, like on an at homes NAS for example. That can be done, but requires manual work and might require more network capabilities and CPU power.

Advanced Networking

Last, but for me the most important point: Having such a box in the private network requires to drill a whole in the firewall, to allow port forwarding. I know, that is nothing unusual for experienced people, and in theory little problem.

But for people who are not so interested, that means they need to click in the interface of their router on a button that they do not understand what it does, and maybe even insert data by following an documentation that they have to believe. (That is not very much different from downloading a script from somewhere letting it do the changes which I would not recommend as well). Doing mistakes here could potentially have a huge impact for the network behind the router, without that the person who did it even has an understanding for.

Also DynDNS is needed: That is also not a big problem in theory and for geeks, but in practice it is nothing easily done.

With a good solution for private cloud, it should not be necessary to ask for that kind of setups.

Where to go from here?

There should be better ways to solve this problems with ownCloud, and I am sure ownCloud is the right tool to solve that problem. I will share some thought experiments that we were doing some time back to foster discussion on how we can use the Raspberry Pi with ownCloud (because it is a very attractive piece of hardware) and solve the problems.

This will be subject of an upcoming blog here, please stay tuned.

There has been the big enterprise release 9.1 on September 20th, but that of course came along with community releases which are in the focus here.

We had server release 8.0.15, server release 8.1.10, server release 8.2.8 and release 9.0.5. There are maintenance releases for the older major versions, needed to fix bugs on installations that still run on these older versions. We deliver them following this plan.

The latest and greatest server release is release 9.1.1 that has all the hardening that also went into the enterprise releases.

Aside a ton of bugfixes that you find listed in the changelog there have also been interesting changes which drive innovation. To pick just one example: The data fingerprint property. It enables the clients to detect if the server got a backup restored, and saves changes on the clients to conflict files if needed. This is a nice example of solutions which are based on feedback from enterprise customers community running ownCloud, who help with reporting problems and proposing solutions.

Talking about professional usage of ownCloud: Of course also all the server release are available as linux packages for various distributions, for example the ownCloud server 9.1.1 packages. We think that our users should not be forced to deploy from tarballs, which is error prone and not native to Linux, but have the choice to use linux packages through the distributions package management.

There also have been client releases recently: The Android client versions 2.1.1 and 2.1.2 were released with important changes for Android 7 and much more fixes, as well as iOS client versions 3.5.0 and 3.5.1. The desktop client 2.2.4 also got a regular bug fix update (Changelog).

I guess you agree that is a lot of activity shown in the ownCloud project, making sure to get the best ownCloud experience out there for the users, driven by passion for the project and professional usage in focus.

If you are interested and want to join in and make ownCloud better, jump in on ownCloud Central or Github. It’s fun!

But there is even more opportunity: Additionally we are looking for an ownCloud Desktop Client developer. That would be somebody fluid in C++ and Qt who likes to pick up responsibility for our desktop client together with the other guys on the team. Shifted responsibilities have created this space, and it is your chance to jump into the desktop sync topic which makes ownCloud really unique.

The role includes working with the team to plan and roll out releases, coordinate with the server- and mobile client colleagues, nail out future developments, engage with community hackers and help with difficult support cases. And last but not least there is hacking fun on remarkable nice Qt based C++ code of our desktop client, together with high profile C++ hackers to learn from.

It is an ideal opportunity for a carer type of personality, to whom it is not enough to sit in the basement and only hack, but also to talk to people, organize, and become visible. Having a Qt- and/or KDE background is a great benefit. You would work from where you feel comfortable with as ownCloud is a distributed company.

The ownCloud Client is a very successful part of the ownCloud platform, it has millions of installations out there, and is released under GPL.

If you want to do something that matters, here you are! Send your CV today and do not forget to mention your github account :-)

What we do - what you will work on

The call is for people who understand the vision of bringing the idea ownCloud to an enterprise ready level: ownCloud is not only running on individual open source enthusiasts hardware, but also on sites with huge amounts of data like CERN or the Sciebo project, and at large companies who want to work with their data in a secure way.

To provide the best solution for all of them we are looking for:

A System Administrator

In this role, you make sure that the infrastructure that we use in ownCloud is up and running. That involves troubleshooting and streamlining existing infrastructure, but also designing new services. If you love virtualization of all kinds and have an eye for security, this position is for you. Of course all this does not only happen behind closed doors, but you will be in contact with the open source community around ownCloud.

The Application Security Engineer

For security professionals who would like to take on a high profile open source project. As security is one of the core values of ownCloud, we are looking for somebody who constantly monitors the code flowing in for security problems, is able to find glitches in existing code and handle the bug bounty program. That and more is the task of this high profile position.

A Software Engineer PHP

For engineers with a passion for good software design and a love for writing code without being code monkeys: In this role you iron the server part of our platform, build new features, work on fixing bugs with the support colleagues and bother the architect with new ideas how to make the thing even better. For this you need to urge to get down and dirty with code, feel yourself comfortable in a team of high profile developers who can teach you things and learn from you.

PHP or what?

Yes, ownCloud is written in PHP, and PHP is the most important, but by far not the only language that we use for the ownCloud platform.

Before you turn your back because of PHP, please think twice. There are a lot of good reasons why we are going with PHP, some of them are named in this blog, but there is more: For example PHP7: With PHP 7 (which can be used with ownCloud) the language has caught up with many criticism it faced before and has done a big leap.

And anyway, the language of a system is not the only thing that is important in a developers life. It is rather how many people use, love and recommend the project and the development processes the team lives. And in all that points, ownCloud is already awesome, and will become even more with your help.

Send your resume in to work@owncloud.com so we can get talking!

Overlay Icons

Overlay icons for the various file managers on our three platforms already exist for quite some time, but it has turned out that the performance was not up to the mark for big sync folders. The reason was mainly that too much communication between the file manager plugin and the client was happening. Once asked about the sync state of a single file, the client had to jump through quite some hoops in order to retrieve the required information. That involved not only database access to the sqlite-based sync journal, but also file system interaction to gather file information. Not a big deal if it’s only a few, but if the user syncs huge amounts, these efforts do sum up.

This becomes especially tricky for the propagation of changes upwards the file tree. Imagine there is a sync error happening in the foo/bar/baz/myfile. What should happen is that a warning icon appears on the icon for foo in the file manager, telling that within this directory, a problem exists. The complexity of the existing implementation was already high and adding this extra functionality would have reduced the reliability of the code lower than it already was.

Jocelyn was keen enough to do a refactoring of the underlying code which we call the SocketApi. Starting from the basic assumption that all files are in sync, and the code has just to care for these files that are new or changed, erroneous or ignored or similar, the amount of data to keep is very much reduced, which makes processing way faster.

Server Notifications

On the ownCloud server, there are situation where notifications are created which make the user aware of things that happened.

An example are federated shares:

If somebody shares a folder with you, you previously had to acknowledge it through the web interface. This explicit step is a safety net to avoid people sharing tons of Gigabytes of content, filling up your disk.

With 2.2.x, you can acknowledge the share right from the client, saving you the round trip to the web interface to check for new shares.

Keeping an Eye on Word & Friends

Microsoft Word and other office tools are rather hard to deal with in syncing, because they do very strict file locking of the files that are worked on. So strict that the subsequent sync app is not even allowed to open the file, not even for reading. That would be required to be able to sync the file.

As a result the sync client needs to wait until word unlocks the file, and then continue syncing.

For previous version of the client, this was hard to detect and worked only if other changes happened in the same directory where the file in question resides.

With 2.2.0 we added a special watcher that keeps an eye on the office docs Word and friends are blocking. And once the files are unlocked, the watcher starts a sync run to get the files to the server, or down from the server.

Advances on Desktop Sharing

The sharing has been further integrated and received several UX- and bugfixes. There is more feedback when performing actions so you know when your client is waiting for a response from the server. The client now also respect more data returned from the server if you have apps enabled on the server that for example limit the expiration date.

Further more we better respect the share permissions granted. This means that if somebody shared a folder without create permissions with you and you want to reshare this folder in the client you won’t get the option to share with delete permissions. This avoids errors when sharing and is more in line with how the whole ownCloud platform handles re-sharing. We also adjusted the behavior for federated reshares with the server.

Please note to take full advantage of all improvements you will need to run at least server version 9.0.

Have fun!

With the processing described in the first part of the blog, the upload is done savely and with a clean approach, but it also has some drawbacks.

Most notably the server does not know the target filename of the uploaded file upfront. Also it does not know the final size or mimetype of the target file. That is not a problem in general, but imagine the following situation: A big file should be uploaded, which would exceed the users quota. That would only become an error for the user once all uploads happened, and the final upload directory is going to be moved on the final file name.

To avoid useless file transfers like that or to implement features like a file firewall, it would be good if the server would know these data at start of the upload and stop the upload in case it can not be accepted.

To achieve that, the client creates a file called _meta in /uploads/ before the upload of the chunks starts. The file contains information such as overall size, target file name and other meta information.

The server’s reply to the PUT of the _meta file can be a fail result code and error description to indicate that the upload will not be accepted due to certain server conditions. The client should check the result codes in order to avoid not necessary upload of data volume of which the final MOVE would fail anyway.

This is just a collection of ideas for an improved big file chunking protocol, nothing is decided yet. But now is the time to discuss. We’re looking forward to hearing your input.

The third and last part will describe how this plays into delta sync, which is especially interesting for big files, which are usually chunked.

This is the first of three little blog posts in which we want to present the idea and get your feedback. This is for open discussion, nothing is set in stone so far.

What is the downside of the current approach? Well, the current algorithm needs a lot of distributed knowledge between server and client to work: The naming scheme of the part files, semi secret headers, implicit knowledge. In addition to that, due to the character of the algorithm the server code is too much spread over the whole code base which makes maintaining difficult.

This situation could be improved with the following approach.

To handle chunked uploads, there will be a new WebDAV route, called remote.php/uploads. All uploads of files larger than the chunk size will go through this route.

In a nutshell, an upload of a big file will happen as parts to a directory under that new route. The client creates it through the new route. This initiates a new upload. If the directory could be created successfully, the client starts to upload chunks of the original file into that directory. The sequence of the chunks is set by the names of the chunk files created in the directory. Once all chunks are uploaded, the client submits a MOVE request the renames the chunk upload directory to the target file.

Here is a pseudo code description of the sequence:

1. Client creates an upload directory with a self choosen name (ideally a numeric upload id):

MKCOL remote.php/uploads/upload-id

2. Client sends a chunk:

PUT remote.php/uploads/upload-id/chunk-id

3. Client repeats 2. until all chunks have successfully been uploaded 4. Client finalizes the upload:

MOVE remote.php/uploads/upload-id /path/to/target-file

5. The MOVE sends the ETag that is supposed to be overwritten in the request header to server. Server returns new ETag and FileID as reply headers of the MOVE.

During the upload, client can retrieve the current state of the upload by a PROPFIND request on the upload directory. The result will be a listing of all chunks that are already available on the server with metadata such as mtime, checksum and size.

If the server decides to remove an upload, ie. because it hasn’t been active for a time, it is free to remove the entire upload directory and return status 404 if a client tries to upload to. Also, a client is allowed to remove the entire upload directory to cancel an upload.

An upload is finalized by the MOVE request. Note that it’s a MOVE of a directory on a single file. This operation is not supported in normal file systems, but we think in this case, it has a nice well descriptive meaning. A MOVE is known as an atomic and fast operation, and that way it should be implemented by the server.

Also note that only with the final MOVE the upload operation is associated with the final destination file. We think that this approach already is a great improvement, because there is always a clear state of the upload with no secret knowledge hidden in the process.

In the next blog I will discuss an extension to this that adds more features to the process.

What do you think so far? Your feedback is appreciated, best on the ownCloud devel mailinglist!

[caption id=“attachment_586” align=“alignleft” width=“300”] The Share Dialog[/caption]This release brings a new integration into the operating system file manager. With 1.8.0, there is a new context menu that opens a dialog to allow the user to create a public link on a synced file. This link can be forwarded to other users who get access to the file via ownCloud.

Also the clients behavior when syncing files that are opened by other applications on Windows has greatly been improved. The problems with file locking some users saw for example with MS office apps were fixed.

Another area of improvements is again performance. With latest ownCloud servers, the client uses even more parallized requests, now for all kind of operations. Depending on the synced data structure, this can make a huge difference.

All the other changes, improvements and bug-fixes are too hard to count. Finally, this release received around 700 git commits compared to the previous release.

All this is only possible with the powerful and awesome community of ownClouders. We received a lot of very good contributions through the GitHub tracker, which helped us to nail down a lot of issues and improved the client tremendously.

But this time we’d like to specifically point out the code contributions of Alfie “Azelphur” Day and Roeland Jago Douma who contributed significant code bits to the sharing dialog on the client and also some server code.

A great thanks goes out to all of you who helped with this release. It was a great experience again and it is big fun working with you!

We hope you enjoy 1.8.0! Get it from https://owncloud.org/install/#desktop

ETags

ETags are strings that describe exactly one specific version of a file (example: 71a89a94b0846d53c17905a940b1581e).

Whenever the file changes, the ownCloud server will make sure that the ETag of the specific file changes as well. It is not important in which way the ETag changes, it also does not have to be strictly unique, it’s just important that it changes reliably if the file changes for whatever reason. However, ETags should not change if the file was not changed, otherwise the client will download that file again.

In addition to that, The ETags of the parent directories of the file have to change as well, up to the root directory. That way client systems can detect changes that happen somewhere in the file tree. This is in contrast to normal computer file systems where only the modification time of the direct parent of a file is changing.

File IDs

FileIDs are also strings that are created once at the creation time of the file (example: 00003867ocobzus5kn6s).

But contrary to the ETags, the file IDs should never ever change over the files lifetime. Not on an edit of the file, and also not if the file is renamed or moved. One of the important usages of the FileID is to detect renames and moves of a file on the server.

The FileID is used as an unique key to identify a file. FileIDs need to be unique within one ownCloud, and in inter-owncloud connections, they must be compared together with the ownCloud server instance id.

Also, the FileIDs must never be recycled or reused.

Checksums?

Often ETags and FileIDs are confused with checksums such as MD5 or SHA1 sums over the file content.

Neither ETags nor FileIDs are, even if there are similarities: Especially the ETag can be seen as a checksum over the file content. However, file checksums are way more costly to compute than just a value that only needs to change somehow.

What happens if…?

Let’s make a thought experiment and consider what it would mean especially for sync clients if either fileID or ETag gets lost from the servers database.

If ETags are lost, clients loose the ability to decide if files have changed since the last time that was checked by the clients. So what happens is that the client will download the files again, byte-wise compare them to the local file and use the server file if the files differ. A conflict file will be created. Because the ETag was lost, the server will create new ETags on download. This could be improved by the server creating more predictable ETags based on the storage backends capabilities.

If the ETags are changed without reason, for example because a backup was played back on the server, the clients will consider the ones with changed ETags as changed and redownload them. Conflict handling will happen as described if there was a local change as well.

For the user, this means a lot of unnecessary downloads as well as potential conflicts. However, there will not be data loss.

If FileIDs got lost or changed, the problem is that renames or moves on server side can no longer be detected. That would result in a new download of files in the good case. If a fileID however changes to something that was used before, that can result in a rename that overwrites an unrelated file. That is because clients might still have the FileID associated with another file.

Hopefully this little post explains the importance of the additional metadata that we maintain in ownCloud.

RSync is great

Nobody will object here. In a nutshell, this is how rsync works: There is a file on the client and on the server. The idea is to not transfer the entire file from one side to the other if either side changes, but only the parts that have changed.

The original rsync does that by chopping the file to blocks of a given size and calculating a checksum of each of the blocks. The list of checksums is sent to the server and - here’s the trick - the server looks at its version of the file and for each of the checksum in the list, it seeks if it finds the same block in the file. That will often not be at the same position in the file, but maybe somewhere else. That is done for each block, and finally the server will work out the information of which parts of the file are existing and which are not and have to be sent by the client.

By way of this clever algorithm, we will just have to transmit a very small fraction of the changed file, because most content did not change. And that is what we want! Yeah!

Mission accomplished? No, not really. While there is basically nothing wrong with the idea in general, there is a severe architectural downside. The rsync algorithm depends on a strong server component which, for each file, searches around and calculates checksums. In an environment where we potentially have a lot of clients connecting to one server that would create a huge load on it which we need to avoid. So what if instead of putting the burden on the server’s shoulder, we could make the clients take the responsibility?

And guess what, there has been somebody thinking about that before and he says:

Use ZSync for this!

ZSync basically turns the idea of rsync upside down and shifts the calculation of checksums away from the server and onto the clients. That means that with zsync, the server can keep a static list of checksums for every block specific to a version of a file. The list can for example be computed along the upload of the file to the server. From that point it does not change, as long as the file does not change. That means less computation work for the server, and maybe this job can also put into the client.

So far that sounds cool (even though some questions remain) and sounds like something that can help us.

Unfortunately, the approach does not work very well for compressed files. The reason is that if a file gets compressed, even if only a couple of bytes in the original file change, the compression algorithm usually changes a lot all over the entire file. As a result, the zsync algorithm can only compute a comparably large diff. Given the cost of computation that can turn inefficient quickly.

“But who uses compressed files?” you might argue. The problem is that almost each and every of the files in everyday life are stored compressed. This is for example true for Microsoft Office files and the Open Document files produced by LibreOffice and Apache OpenOffice. They are really renamed ZIP containers, that hold the documents with all its embedded files, etc.

Now of course you will reply that zsync has an improved algorithm for compressed files. Yes, true, that is a great thing. However, it involves that the compressed file gets uncompressed to be worked on by zsync. Afterwards it is compressed again. And that is the problem: As common compressors do not leave a hint behind _how_ the file was compressed, it is not possible to reliably recreate a file that is equivalent to the original one. How will apps react on a file that has changed its compression scheme?

Results

As said above, yes, we will at one point of time implement something along the zsync algorithm. The explanations above should show however, that at the current state of ownCloud, other features will improve ownClouds performance, stability and convenience more. And that is the important thing for us, more than pleasing the loudest barking dogs. Here is a rough outline of how I would move on on this, open for your suggestions and critique: The zsync algorithm is designed to improve downloads. We need it for both up- and downloads, and it needs to be thought through if that is also possible. For the server side functionality, there are a couple of open questions which carefully have to be investigated. Preferably an app can be written that provides the handling of the zsync checksum lists. That has to be clarified and discussed, and that will take a while. But as outlined above, this idea is only clever for a limited amount of file types. So what I would suggest first is that we get an idea of the file types users usually store in their ownCloud, so that we can do a validated estimate on how this feature helps. I will follow up on this first step.

Thanks for reading this long blog post. Thanks Danimo for lectorate.

We prepared some test packages in our development repository isv:ownCloud:devel for those who wanna try and know their way around. Current it only builds for a couple of openSUSE Distros. You need to install kdebase4 and dolphin-plugins and after installation, it’s easiest to restart KDE to make it registered. But be warned: The two packages replace packages from the previous installation, only do it if you really know what you’re doing!

It would be great if at least the libkonq patch could make it to upstream, and I would appreciate if somebody who is a bit more fluent with recent KDE libs development could give me a hand on that. Otherwise, if distros wanna pick up the patches to make the overlays work, of course the patches are here: patch for libkonq and the ownCloud Dolphin plugin. The plugin will work with the released version 1.7.0 of ownCloud Client.

CERN is a very inspiring place, attracting intelligent people from all over the world to get behind the secrets of our being. I felt honored to be at the place where for example the world wide web was invented.

The event was called Workshop on Cloud Services for File Synchronisation and Sharing and was hosted by CERN IT department. There have been around 100 attendees.

I was giving a talk called The File Sync Algorithm of the ownCloud Desktop Clients, which was very well received. If you happen to be interested in the sync algorithm we’re using, the slides are a nice starting point.

What amazed me most was the great atmosphere and the very positive attitude towards ownCloud. Many representatives of edu organizations that use ownCloud to which I talked were very happy with the product (even though there are problems here and there) from the technical POV. A lot of interesting setups and environments were explained and also showcased ownCloud’s flexibility to integrate into existing structures.

What also was pointed out by the attendees of the workshop was the importance of the fact that ownCloud is open source. Non free software does not have a chance at all in that market. That was the very clear statement in the final discussion session of the workshop.

The keynote was given by Prof. Benjamin Pierce from Pennsylvania with the title Principles of Synchronization. He is the lead author of the project Unison which is another opensource sync project. It’s sync engine marks very high quality, but is not “up-to-date software” any more as he said. I had the pleasure to spend quite some time with him to discuss syncing in general and our sync algorithms in particular, amongst other interesting things.

[caption id=“attachment_525” align=“alignleft” width=“254”] Atlas Detectors[/caption]As part of his work, he works with a tool called QuickCheck to do very enhanced testing. One night we were sitting in the cantina there hacking to adopt the testing to ownCloud client and server. The first results were very promising, for example we revealed a “problem” in our sync core that I knew of, which formally is a sync error, yet very very unlikely to happen and thus accepted for the sake of an easier algorithm. It was impressive how fast the testing method was identifying that problem. I like to follow up with the testing method.

Furthermore we met with a whole variety of other interesting people, backend developers, operators of the huge datasets (100 Peta-Byte), the director of CERN IT, a maintainer of the Scientific Linux and others.

Also we had the chance to visit the Atlas experiment, it is 100 meter underneath the surface and huge. That is where the particles are accelerated, and it was great to have the chance to visit that.

The trip was a great experience and very motivating for me, and I think it should be for all of us all doing ownCloud. Frank was really hitting a nerv when he was seeding the idea, and we all were doing a nice product of it so far.

Lets do more of this cool stuff!

The new release brings two lighthouse features which I’ll briefly describe here.

Overlay icons

For the first time, this release has a feature that lives kind of outside the ownCloud desktop client program. That nicely shows that syncing is not only a functionality living in one single app, but a deeply integrated system add-on that affects various levels of desktop computing.

Here we’re talking about overlay icons which are displayed in the popular file managers on the supported desktop platforms. The overlay icons are little additional icons that stick on top of the normal file icons in the file manager, like the little green circles with the checkmark on the screenshot.

The overlays visualize the sync state of each file or directory: The most usual case that a file is in sync between server and client is shown as a green checkmark, all good, that is what you expect. Files in the process of syncing are marked with a blue spinning icon. Files which are excluded from syncing show a yellow exclamation mark icon. And errors are marked by a red sign.

What comes along simple and informative for the user requires quite some magic behind the curtain. I promise to write more about that in another blog post soon.

Selective Sync

Another new thing in 1.7.0 is the selective sync.

In ownCloud client it was always possible to have more than one sync connection. Using that, users do not have to sync their entire server data to one local directory as with many other sync solutions. A more fine granular approach is possible here with ownCloud.

For example, mp3’s from the Music dir on the ownCloud go to the media directory locally. Digital images which are downloaded from the camera to the “photos” dir on the laptop are synced through a second sync connection to the server photo directory. All the other stuff that appears to be on the server is not automatically synced to the laptop which keeps it organized and the laptop harddisk relaxed.

While this is of course still possible we added another level of organization to the syncing. Within existing sync connections now certain directories can be excluded and their data is not synced to the client device. This way big amounts of data can be easier organized depending on the demands of the target device.

To set this up, check out for the button Choose what to Sync on the Account page. It opens the little dialog to deselect directories from the server tree. Note that if you deselect a directory, it is removed locally, but not on the server.

What else?

There is way more we put into this release: A huge amount of bug fixes and detail improvements went in. Fixes for all parts of the application: Performance (such as database access improvements), GUI (such as detail improvements for the progress display), around the overall processing (like how network timeouts are handled) and the distribution of the applications (MacOSX installer and icons), just to name a few examples. Also a lot of effort went into the sync core where many nifty edge cases were analyzed and better handled.

Between version 1.6.2 and the 1.7.0 release more than 850 commits from 15 different authors were pushed into the git repository (1.6.3 and 1.6.4 were continued in the 1.6 branch which commits are also in the 1.7 branch). A big part of these are bug fixes.

Who is it?

Who does all this? Well, there are a couple of brave coders funded by the ownCloud company working on the client. And we do our share, but not everything. Also coding is only one thing. If you for example take some time and read around in the client github repo it becomes clear that there are so many people around who contribute: Reporting bugs, testing again and again, answering silly looking questions, proposing and discussing improvements and all that (yes, and finally coding too). That is really a huge block, honestly.

Even if it sometimes becomes a bit heated, because we can not do everything fast enough, that still is motivating. Because what does that mean? People care! For the idea, for the project, for the stuff we do. How cool is that? Thank you!

Have fun with 1.7.0!

So I tried to investigate the app with Qt Inspector of Robert Knight. That’s a great tool, but it does not yet completely do what I need because it only shows QWidget based objects. But Robert was kind enough to put me on the right track, thanks a lot for that!

I tried this naive approach:

In the clients main.cpp, I implemented these both callback functions:

[code language=“cpp”] QSet<QObject*> mObjects;

extern “C” Q_DECL_EXPORT void qt_addObject(QObject *obj) { mObjects.insert(obj); }

extern “C” Q_DECL_EXPORT void qt_removeObject(QObject *obj) { mObjects.remove(obj); } [/code]

Qt calls these callbacks whenever a QObject is created or deleted respectively. When the object is created I add it’s pointer to the QSet mObjects, and if it is deleted, it is removed from the QSet. My idea was that after the QApp::exec() call returns, I would have to see which QObjects are still in the mObjects QSet. After a longer run of the client, I hoped to see an artificial amount of objects being left over.

Well, what should I say… No success so far: After first tests, it seems that the amount of left-over objects is pretty constant. Also, I don’t see any objects that I would not kind of expect.

So this little experiment left more questions than answer: Is the suspicion correct that QObjects with a valid parent pointer can cause the memory growth? Is my test code as I did it so far able to detect that at all? Is it correct to do the analysis after the app.exec() call returned?

If you have any hints for me, please let me know! How would you tackle the problem?

Thanks!

This is the link to my modified main.cpp: https://github.com/owncloud/mirall/blob/qobject_monitor/src/main.cpp

In my opinion, ownCloud 7 is a real step ahead. So much work went into it from the brave server guys, and the result is simply impressive: It is fast, beautiful and fully focused on what it is supposed to be and do: File sync and share, even though you still have of course all the other useful stuff like calendaring and such.

Apart from the wellknown features ownCloud 7 also brings some things that you do not see on the first look. Some of these things relate to syncing and as a guy working mainly on the sync client I am especially happy about that: Already with the current released clients you will see a performance win in syncing, because ownCloud 7 handles parallel HTTP requests way better. Moreover, ownCloud 7 sends some interesting information about it’s files, and the next generation of sync clients will for example know which files are shared and their permissions. We are currently working on some exciting stuff on the client side, stay tuned.

That all should be reason enough to celebrate together, because ownCloud is done by all of us in the community.

On Tuesday, august 5th, we will do a Release Party in Nuremberg, Germany. As last time, the Coworkingspace Nürnberg is hosting us. We will start at 18:00.

Everybody who is interested in ownCloud for whatever reason is very welcome to show up. We will probably have a short informal demonstration or two, but of course plenty room for questions, discussions, meet and greet with all kind of FOSS people, some core ownCloud people, nice beer and big fun.

You really should come! And don’t forget to tell your friends, thanks!

The recommendation is to update your installation to this version. The previous version 1.6.0 had great new features, first and foremost the parallel up- and download of files and a way more performant handling of the local sync journal. That required a lot of code changes. Unfortunately that also brought in some bugs which are now fixed with the 1.6.1 release.

On the windows platforms, we experienced a memory leak that over time let the memory consumption of the client grow. Also, a problem in the Qt5 library that we ship for windows caused the problem that under some circumstances the wrong encryption lib was loaded, so that some people saw SSL problems on Windows.

And there were crashes. Users kept on reporting that the client was crashing after some time on windows, without a special reason. None of the developers was able to reproduce that or ever saw that. We asked for backtraces, which also can be produced on windows. Even though the backtraces looked similar, we did not find an obvious reason for the crashes. Finally, by reading through all involved code levels again and again, Olivier was able to spot some code in libneon that, again under special circumstances, could cause crashes on win.

It was a one line fix, we quickly built test packages, people tested, and finally the crashes were gone (the patch to libneon is on its way to upstream of course).

All that is now fixed in 1.6.1.

What does that show? There not very much little “easy” bug findings any more. That is similar to the soccer world championship, where the coaches keep telling that there are no “easy opponents” any more nowadays, which is also true. These tricky problems we face in the client are hard to find, require time, often special setups if they are reproduceable at all, and advanced debugging skills. Very challenging, very much fun. But that also requires very much patience from the people who suffer from that bugs. We keep on asking questions, ask to test new daily builds and need time to investigate stuff, and more time to release once we have the fix.

Thank you all for helping in this situation, not giving up, for again testing another daily build, reporting back, trying again. That is really big.

Even on saturday morning, short after opening of the event, somebody from the orga team was showing up on the booth with chocolate for the volunteers, saying hello and asking if everything is in place for a successful weekend. A small detail, which shows how much effort is put into organization of the event.

As a result, visitors come to visit the event. It’s mostly a community centric event: Exhibitors are mostly representing FOSS projects such as openstreetmap.org, distributions like Fedora or openSUSE or companies from the free software market. [caption id=“attachment_428” align=“alignright” width=“595”] Morris in action on the booth[/caption]The majority of visitors are mostly interested in private use of the software. But, no rule without exception, we also had a remarkable number of people from companies, either executives or people working in the IT departments, who were interested in ownCloud.

Speaking about ownCloud, I want to say that it’s amazing to represent our project. People know it, people like it, people use it. In private, but also in professional space people work with ownCloud already or are planing to start with ownCloud. ownCloud already is the accepted solution for the problems that became so practical with the NSA scandal last year.

My talk with title A private Cloud with ownCloud on Saturday morning was very well received and went smooth. The room was too small, lots of people had to stand or sit on the stairs. It was a very positive atmosphere.

Something that changed compared to last year and the year before: Most discussions were around how ownCloud can be installed, integrated and used and not any more about which features are still missing or maybe also bugs.

So it were two very exhausting days, but big fun! Thanks to Daniel, Arthur and Morris for the work and fun we had on the booth, and thanks to the CLT team for CLT.

The ownCloud community released ownCloud 6 a couple of days ago. That was another big release and we want to celebrate!

Please, everybody who is interested in ownCloud, like to learn more, give feedback or just want to meet other people from the community, you are invited to show up at Coworking Space in Nuremberg, Josephsplatz 8, on december 18th, 6pm.

We will have a relaxed evening with a little discussion, maybe short demos, cakes and stuff, and fun. No heavy talks and serious faces!

We are looking forward to meeting you.

To do a step into that direction, here is a little script collection for you to play with if you like: the DAV torture collection. We started it quite some time ago but never really introduced it. It is still very rough.

What it does

The first idea is that we need a reproducable set of files to test the server with. We don’t want to send around huge tarballs with files, so Danimo invented two perl scripts called torture_gen_layout.pl and torture_create_files.pl. With torture_gen_layout.pl one can create a file that contains the layout of the test file tree, a so called layout( or .lay)-file. The .lay-file describes the test file tree completely, with names, structure and size.

torture_gen_layout.pl takes the .lay-file and really creates the file tree on a machine. The cool thing about is that we can commit on a .lay-file as our standard test tree and just pass a file around with a couple of kbytes size that describes the tree.

Now that there is a standard file tree to test with, I wrote a little script called dav_torture.pl. It copies the whole tree described by a .lay file and created on the local file system to an ownCloud WebDAV server using PUT requests. Along with that, it produces performance relevant output.

Try it

Download the tarball and unpack it, or clone it from github.

After having installed a couple of perl deps (probably only modules Data::Random::WordList, HTTP::DAV, HTTP::Request::Common are not in perl’s core) you should be able to run the scripts from within the directory.

First, you need to create a config file. For that, copy t1.cfg.in to t1.cfg (don’t ask about the name) and edit it. For this example, we only need user, passwd and url to access ownCloud. Be careful with the syntax, it gets sourced into a perl script.

Now, create the local reference tree with a .lay-file which I put into the tarball: ./torture_create_files.pl small.lay tree This command will build the file tree described by small.lay into the directory called tree.

Now, you can already treat your server: Call ./dav_torture.pl small.lay tree This will perform PUT commands to the WebDAV server and output some useful information. It also appends to two files results.dat and puts.tsv. results.dat just logs the results of subseqent call. The tsv file is the data file for the html file index.html in the same directory. That opened in a browser gives a curve over the average transmission rate of all subsequent runs of dav_torture.pl (You have run dav_torture.pl a couple of times to make that visible). The dav_torture.pl script can now be hooked into our Jenkins CI and performed after every server checkin. The resulting curve must never raise :-)

To create your own .lay-file, open torture_gen_layout.pl and play with the variables on top of the script. Simply call the script and redirect into a file to create a .lay-file.

All this is pretty experimental, but I thought it will help us to get to a more objective discussion about performance. I wanted to open this up in a pretty early stage because I am hoping that this might be interesting for somebody of you: Treat your own server, create interesting .lay files or improve the script set (testing plain PUTs is rather boring) or the result html presentation.

What do you think?

You find suitable download links as usual at: http://owncloud.org/sync-clients/

Version 1.4.1 is a bugfix release for the 1.4.0 version released a few of weeks ago which brought a lot of new features. This one solves a couple of problems that were coming up during the last few weeks. For example, the problem that the client lost its configuration (at least) on the Win32 platform when the machine was shut down is fixed. Also a lot of redundant uploads wont happen any more. And there are even more fixes, as the detailed Changelog rules out.

We thank you for your ongoing support and the good work on bugs that came up. As usual we are looking forward to your feedback. Please work with us in the Github bugtracker if you experience issues.

Danimos Visual Guide has outlined the new stuff in the release already, so no need to repeat it here. You should install and try it, that seems to be the opinion of many people who tried it.

Also people who shared their critical view on the client very publically in the past are much more pleased now with 1.4.0. One example is a recent blog post on BITBlokes. It is a blog about all kind of topics around FOSS. I regularly read it and often share its opinions. He concludes very positively about the 1.4.0 client.

It is good to see the positive feedback overall. That shows a couple of things from my engineering point of view: The concentrated work we continously do on all parts of ownCloud pays off. That is obvious of course, but still nice to see. And our (also obvious) actions to improve code quality such as the consequent use of continous integration, code reviews and such helps to improve quality.

“People are always excited if releases come with GUI changes!” I heard people saying. Well, maybe, but that’s not the whole truth. It also proves for me again is how important UI design and UX is. Me as a knee-deep-developer have an interesting relationship to all UX topics: I always have an opinion. Often a strong opinion. But the results coming out of that have not always been the, well, the most optimal. Very fortunate on the client we work together with our UX guy Jan and the positive feedback also shows how good that is for the software.

But enough of release pride. There is more work to do: The bug tracker is still not empty, the list of feature ideas is long. We will continue to focus on correctness, stability and robustness of syncing, performance and useful features and work on a version 1.5 for you.

These are a couple of concrete points we’re focussing on for 1.5:

1. we already merged the client code on the new upstream sync version in git.
2. performace improvements through further reduction of the number of requests and more efficiency in database operations on the client.
3. we are working on a new propagator component that allows us to do the changes mentioned in 2 more easily.
4. File manager integration, which means havingn icons in Explorer, Dolphin and friends.

A more detailed list can be found at github.

Thank you for all your help and support. It’s big fun!

Many users of the ownCloud Client were asking for sync progress information, in fact there was none at all until today which is a bit boring. The reason why we hadn’t it was simply that csync, which is the file synchronizer engine we use, did not have an API to hand over progress information of an actual up- or download to higher levels of the application.

We implemented two callbacks in csync: One that informs about start, end and progress of an up- or download of an individual file. Another one processes the overall progress of the currently running sync run, indicating for example that eight files have to be processed, current is file number four, and x of the overall sum y bytes have been processed already.

That information is passed into a singleton class called ProgressDispatcher in the client code, where other classes can connect to a signal providing that information. That comes handy as we need the information at different places in the client GUI. [caption id=“attachment_299” align=“alignright” width=“399”] Sync Progress Display[/caption] The screenshot shows the first and probably most detailed implementation of the progress display in the sync accounts details which are part of a new settings- and status dialog.

The visual appearance was worked out by our interaction designer Jan. We always have to argue a bit because obviously I am the greatest interaction designer around ;-) but well, finally we do like Jan says and the result is great IMO. It’s great that we have him for ownCloud as that guarantees that our project is not drifting too much onto the geeky techy side which (I heard) is scaring off some users…

Hope you like it! You can get a preview in the current nightly builds (win, mac, Linux) of the client! Please report bugs as you find them, thanks.

We revisited this problem and will come up with a better solution.

The key changes will probably be

• Conflict files will never again be generated on the server. Even if we are in a conflict situation, we will download the file and keep the conflicting version only on the client. This enables us to detect false conflicts.
• The current way we handle a system time difference has to be changed. We wont adjust the file mtimes of files in the file system any more with the time difference between the client and server. That way we do not suffer from floating time differences any more. For the decision of which version is more recent, we will still consider the time difference.
• We will use a very quick request like OPTION to get the servers time setting to the client. That will allow to calculate the time difference between server and client more accurately. It’s needed to decide which file is more recent.

Read this as a note to self, yet we feel very well fitting into Berlin round may 1st ;-)

Photo credit: 96dpi / CC BY-NC

Here is a small overview of what went into ocsync master (actually the dav branch) and into mirall master.

Note that this is alpha state software, but it’s already available for testing from our nightly builds at the ownCloud nightly repository.

If you enable the nightly package repository on your system, please be sure to use the machine for testing purposes only. Windows and Mac nightly builds can be found here to download.

csync: Big File Chunking

Handling big files over HTTP is difficult, if files grew bigger than a gigabyte, we faced multiple problems. To deal with that, we thought of splitting up big files and transmit the parts to let the server finally assemble the file again once all parts are there.

We wrote a little library to implement that, called httpbf. It is now integrated in the current development branch. You will see files big files split up into parts for transmission to the server. With that there should not be a filesize limit to work with any more, except the limits we have from the operating system etc.

Up- and Download Resume

With big file chunking, we also introduce up- and download resume. If a file transmission was interrupted and is started over again later, only the missing parts are going to be pushed again. However, we’re not yet transmitting only changed chunks for existing files yet, just to answer that question here before it comes up. That will be subject of more enhancements.

Source File Verification

Another problem with big files is that if one gets copied into the synced dir on the local machine, the copy is not finished before the sync acutally starts. That can result in useless transmissions because inconsistent data is synced.

We added code to the httpbf library that checks for the integrity of the source file. When the first chunk is transmitted, it is checked if the file has still the size and mod time from the beginning. If not, the library waits for two seconds and starts over with a check again. That way, it waits until the source file has stabilized. A final check if the source file hasn’t changed during upload is done for every file when the upload is finished.

CSync Context Reuse

There is the very unpleasant problem on some distros (namely Ubuntu and Fedora) that the ownCloud Client wastes file descriptors. We figured that this is because the system libraries on that platforms fail to unload the csync ownCloud module properly.

Version 1.2.5 contains a quick fix for that, but the real fix is to reuse the csync context within mirall and not build it up and down for every sync run as we did until now. So we developed the function csync_commit which commits the results of the sync run to the database and leaves the csync context intact for the next run. That way we just load and unload the ownCloud module on time and not every 30 seconds. The csync patch already went upstream.

Mirall: New Setup Dialog

There is also the new setup dialog in the nightly builds: It is much more simplified compared to the old one, user only needs to add the url and user credentials and thats it. Also, it syncs the whole ownCloud to a local folder, just like other boxes do. That is one of the most often heard enhancement requests, and here we go.

That all I think would qualify for a good 1.3.0 release, but we need more time to fine tune,polish, test and bugfix. But we would appreciate if people could help test. Please report bugs in the client’s bugtracker. We need to find bugs prior the release, and that happens with your help!

Have fun & Thanks a lot for testing!

We just release ownCloud Client 1.2.5, hurrah! It’s a small bug fix release, but it will make more people happy users! It fixes three ugly crashes we found with the help of encouraged bug hunters. One of them was an race between our thread which runs csync and the Qt SSL stack we’re obviously using. That caused a crash which happened after quite some run time, or also never… Hard to reproduce and debug, but finally Danimo and his mighty friends were able to nail that.

The new client can be found through it’s download page, here is the Changelog.

If you find issues, please help us to improve ownCloud by commenting or reporting the bugs on our bugtracker. Thanks!

The nightly builds of the client for Windows and Mac and the source balls can be downloaded from the nightly directory on our download server. They are named after the last release, followed by a date time stamp, which is simply a concatentation of the year, month and day the nightly was built on. For linux we maintain a so called nightly repository in the openSUSE Buildservice which builds for various distributions. That can be added to the linux package management system and that way every morning, an update is there for you. The ownCloud server is built only in the nightly repository from this daily source.

What are nightlies for?

To test ownCloud is a very difficult task. ownCloud is a complex system consisting of variuos parts (anybody thought ownCloud is just a “LAMP web app”?) which supports a huge variety of environments. That multiplies to a huge variety that needs to be tested. We had issues with that in the past and as a result, providing easy access to ready-to-run testing versions is one important action we are taking to improve quality as that enables more people to participate on tests.

Of course nightly builds are not released software, nor got they extensive testing. Nightly builds must never be used on production installations, please, really don’t do.

But maybe you have a test server or virtual machine somewhere in spare. So it would be great if you could install a nightly version on that, combine it with a nightly built client, and see if everything runs as you would expect it.

Another great purpose of the nightly builds is to use it for bug verification. In client development we often comment on bugs with sentences like this is fixed now, please verify with the current nightly build. That way bug reporter get a very quick way to verify the fix of their issue and developers can be sure that their fix really is sufficient.

How are nightlies built?

We are doing the nightly builds through an instance of Jenkins, a continous integration server. With a little script we wrote Jenkins feeds the OBS every night, so this is completely automatted. But more on that in a subsequent blog.

Let’s meet Wednesday, February 27th, 6:00 PM at the Nuremberg Coworking Space.

Talks (short, don’t be afraid!):

• ownCloud overview by Holger Dyroff
• The updated ownCloud 5 User Interface by Jan-Christoph Borchardt
• Next steps for the ownCloud Desktop Clients by Daniel Molkentin

The evening closes with a get-together, your chance to meet the team and discuss in a relaxed atmosphere.

Everybody is welcome!

What is interesting about the release from a more technical point of view? Here are a couple of examples.

One of the things which often was complained about was the performance of the client. Performance is a very broad term, so we have to examine the details: Many people felt like its a performance problem that former clients polled the local file system for changes on the MacOSX and Windows platform. That was recommended from other projects which have experience with going through large file collections. QFileSystemWatcher seemed not to be designed for this usecase.

Polling was ok for desktop computers with up-to-date hardware, but on devices running from batteries this was a energy drain. Good that we could fix that with 1.2.0 using native implementation of change detectors on Windows and MacOSX. They detect changes within a file system tree. If a change is detected, a sync run is started. But note, this is only for local file systems. Detection of changes on the server is a different story which has to be told as well.

Another performance problem was within the file upload which is done by a HTTP PUT request. Month ago (pretty after my start at ownCloud) I implemented it using a tmp file in between. That means, the source file was copied to a tmp file, and from that it was processed to the request body. Improvement was needed and we changed the code to directly read from the file descriptor of the opened source file.

Another thing that was improved with 1.2.0 is the error reporting to the user. Former versions of the client sometimes provided error messages which were not really accurately describing the problem. The reason for that was that csync uses errnos (yes, the ones from errno.h) to name errors as csync maps everything to a POSIX file interface. That surely works as long as you’re on a kind of file system. But it’s hard to map HTTP communication problems onto that. So we decided to add our own “custom” errnos and enhance the whole idea to use these to describe problems. That works more accurate now.

The next things on the list are for example a more convenient setup dialog for the client. Also we will get away from the kind of hard coded target file name on the cloud. A better network recognition will also be next as well as better handling of big files. And more…

Thanks a lot to all who helped to get 1.2.0 finished! It is big fun to work in such a great community :-)

The release of version 1.2.0 is planned for the next week if things go smooth.

[caption id=“attachment_219” align=“alignright” width=“595”] New Sync Protocol Dialog[/caption]

In particular, the the following improvements were added:

• Proxy authentication fixed (Basic auth, NTLM will not yet work)
• The status dialog now provides statistics on the last sync run (via the info button). It will tell in detail which files have been synced, added or deleted.
• Client will go offline while the server in in maintenance mode (feature available with ownCoud master only)
• Improved SSL Certificate acceptance
• All sizes of the new icons are available.
• Support files > 2 GB on all platforms for uploading.
• Fixed some minor memory leaks and again saved some server requests through optimizations.
• Improved error reporting to the user.
• Remove legacy theming support.

We would appreciate if you give this release a test ride. Note that because it is beta you should make extra sure to have have backups of your data.

If you want to give feedback, please use our mailing list for general discussion and the issue tracker for bug reports. Please read our new guidelines on bug reporting before!

Download Links:

• Windows, 32/64 Bit: http://download.owncloud.com/download/testing/owncloud-1.2.0beta2-setup.exe
• Mac OS X ( >= 10.6, 64 bit): http://download.owncloud.com/download/testing/ownCloud-1.2.0beta2.dmg
• Linux: http://software.opensuse.org/download/package?project=isv:ownCloud:testing&package=owncloud-client

Sources:

• Mirall: http://download.owncloud.com/download/testing/mirall-1.2.0beta2.tar.bz2
• OCSync: http://download.owncloud.com/download/testing/ocsync-0.70.1.tar.bz2

Have fun!

This is the first build with the new things we did in Berlin a couple of weeks ago, you will

• discover that there is much better error reporting if something goes wrong.
• probably feel like it syncs faster, yes faster.
• see that there are less HTTP requests to the server for a single sync run.
• don’t see any issues with MacOSX and funny characters in filenames any more.
• recognize a new icon set, which is not finalized yet (actually not all sizes are there, thats why the status dialog looks a bit funny) but we thought its nice to already add it to the beta. It should fit nicely into your operating system environment.
• realize that this client comes with a cross platform file system watcher on clientside, so no polling any more.
• have your password stored in a secure keychain on all platforms since we added qtkeychain to the client.

Maybe there is more, but we thought that’s already a nice beta release.

Please find packages for MacOSX, Windows and Linuxes. Note, not all packages are finished yet. If the one for your distro is missing, please come back later, or even better - speak up at packaging@owncloud.org and help fixing :)

Of course you also should note that this is an early beta and you would not want to use it without a good backup of your data and only on your test account without important data.

We would appreciate if you let us know your experience on the mailinglist. If you find problems, please report it to the client’s bugtracker mentioning client- and server versions and at best with useful logs.

With that we are happily vanishing to spend some time away from the computer, looking back on a very exciting and very busy year, working on an interesting topic with a lot of nice people.

Thanks and best Season’s Greetings!

We thought that it might be fun to work together on csync, our sync engine under the hood of the ownCloud client. There were some issues that should be fixed and on the way we cleaned and improved quite some code in csync.

Here are some things we worked on:

• We added a function that lets the program that uses the csync library pass arbitrary module parameters to the backend module. That way its more easy to steer the behaviour of the ownCloud modules from the calling app.

• Error handling was improved, ie. if an http error happens, csync works errno based error reporting. We added custom errnos because not all error cases with http can be mapped to system errnos.

• Formerly the csync ownCloud module was spooling files through an additional temporar file on client side. That step is skipped now which results in performance improvements as well as in more clean code.

• We were able to reduce the number of HTTP requests that go over the wire even more. For example to check if there are changes on server side, now there is exactly one http propfind required. Also if files have to be synced, we could save some HTTP requests by improving caching of some requests.

• Andreas recently changed the logging system in csync upstream master branch. We merged that back and now do not longer need the log4c framework. One build dependency less and a nice new logging framework.

• Other bugs were fixed, such as a potential crash if a folder as deleted during it is synced, SSL handling shortcomings, code streamlining in handling compressed data streams and more.

• We finalized a patch that uniforms the utf8 representations of characters over all platforms. That will fix problems we saw especially with MacOS and special filenames.

Ah, yes, we also did other things, more related to the ownCloud client. Danimo managed to implement a cross platform filesystem watcher class that is able to fulfill our requirements. That obsoletes polling for changes on the local file system, one of the most popular enhancement requests.

And finally there now is a API in csync thats reports file transmission progress if a callback is installed accordingly. So the client hopefully soon will tell ya what it’s doing for you. Also appreciated I guess…

Last but not least we added code to use QtKeyChain, a cross platform password storage library that stores password encrypted. For example on Linux QtKeyChain connects to kwallet. QtKeyChain was provided by Frank Osterfeld, thanks a lot for that contribution.

Quite some stuff for a short week, note that stuff that fills a short line in this blog can be quite nifty to investigate, implement and test. Not everything is stable, polished and properly integrated but it was a great and productive week. The next release of ownCloud Client will be a nice one.

And sice you can not always work, we had a nice dinner at a very cool italian restaurant. We met with other ownCloud employees located in Berlin, Arthur and Georg. Fun :) And Berlin, yes, a great place to be, but finally I appreciated to arrive back to my snow covered home.

Many thanks to Olivier and Markus for hosting us and for the nice week.

I gave a talk about ownCloud on Saturday where I tried to show some technical details. That could have gone even deeper was the feedback some people gave me so I promise to show more of for example app development next time. Apart from that (and apart from the fact that laptops, projectors and Linux still do not play well) the talk went ok and was well received.

The conference is a joint conference of actually four conferences: The openSUSE Conference, the Czech Linux Days, the Gentoo Summit and the SUSE Labs Conference. Over the weekend it was hosted at the Czech technical university of prague and that was a great venue. Very inspiring that over the whole foyer of the building intersting art sculptures were shown. Unusual for a technical faculty but a great thing. The building is modern and large enough for all the talks and workshops but as a result of that it was sometimes a bit difficult to keep the overview where happens what. Especially because the start- and end times of the talks were not kept in sync over the tracks.

I found it is a different atmosphere on the event compared to the openSUSE conferences of the last years which might have been more focussed around the openSUSE project. This one felt more like an general FOSS event. That is not bad, just different, and since change is obvious and good I look forward to seeing how this influences the openSUSE project in the near future.

I like to thank all involved on the organizing teams for their successful hard work, have a lot of fun for the next two days, I had a lot during the last two days :)

This blog will shed some light on the details. I apologize, it’s a long read.

Time Issues

ownCloud Client versions 1.0.x worked with csyncs traditional way of using the file modification times to detect updates between the two repositories that should be synced to each other. That works fine and conforms to our idea to ideally not use any other metadata in syncing than what the file system has anyway.

However, there is one drawback which we all know from daily life: If at least two parties sync on time its important that all clocks are set exactly the same way. Remember good crime movies where a bank robbery always starts with a clock adjustment of all gangsters? We have exactly the same in ownClouds syncing: All involved have to have the same time setting, otherwise modification times of files can not be compared reliably.

There are solutions for computers to set the exact time (like ntp) so in general that works. However, in real life scenarios these are not reliable because either people do not have them started on the system or because the daemon updates the time once in a while and in that time span the clock skews already too much.

Users all the time reported problems with that and other experts continued to advise that we never get around that problems if we don’t change something fundamental and go away from pure time based syncing.

Well, we did that with our csync version 0.60.0 which is the sync engine for ownCloud Client 1.1.0.

An Unique Id

Now, every file and directory inside a sync directory has an unique Id associated. The idea is that the Id changes if the file changes. So in the sync process the need for a file update in either direction can be computed by comparing the two Ids of the file. If the id has changed on one repository the file was changed there and needs to be synced to the other side.

The Ids are generated on the ownCloud server and one challenge for the client is to always download the correct Id of a file. The Ids are just random tags for a file version. It is not associated to the file content as MD5 sums would be. Actually it was a frequent advise to use MD5 sums or a similar approach which digests the files content to detect updates. That would have come very handy because that means comparing file contents directly and, more important, it’s reproducable on either side. Also the client would have been able to recalculate the MD5-Sum of the local files and would not have depended on a local database with Ids that were pulled from the server before.

But we decided against hashes. Calculating MD5-Sums is costly in terms of CPU and time, especially for large files. The CPU problem is small on clients, but not on servers where a lot of clients connect to. Even though the sums can be calculated during upload, the problems remain for the case where the server does not see the upload stream, think of the “mount my Dropbox” case.

For files on the ownCloud server, the Id is always updated when the file gets updated. On the client side the last Id of a file is in the client database. It is invalidated in case the files modification time changed meanwhile to detect local changes.

Change Propagation

Another remarkable change in the 1.1.0 client is that change events in the file tree propagate up to the top directory on the owncloud server, ie. if a file changes in a directory, the id of the directory changes as well as the one of its parent directory etc.

That means that to detect if a file tree has changed, it’s enough to check the top most directories Id. If that has changed, ok, than the client needs to dig deeper, but in the not so rare case that nothing has changed, the one call is enough to detect that. That dramatically lowers the server load with clients because instead of digging through the whole directory structure what we did with the 1.0.x series it is a few requests now.

CSync and ownCloud for Success

These are very intrusive changes to csync. For example, we had to add two additional fields to the database, add code that is able to build a representation of the local file tree from the database and make csync query for the file Ids from the server if needed. Deep under the hood the updater, reconciler and propagator code needed changes to work with the Ids. All these changes did not go back to csync upstream yet.

To not conflict with the upstream version of csync we decided to rename our csync version to ocsync. But: This is a temporar solution for the time we need to catch up with upstream again. That will take a while until everything is sorted again but we will work on that.

I am are very excited about the new version of csync. But obviously there are other changes in the ownCloud Client 1.1.0 which will be subject of another blog post.

The OBS is a great system as it does not only build for various Linuxes but also lets the users download the binary packages utilizing a huge mirroring infrastructure. All for you, all for the good of free software.

And there is even more: OBS offers a nice download page which can either be embedded into a projects webpage or be linked from it. Depending on the settings of the OBS project, for example the number of Linuxes it builds for, the page is adopted automatically. So you never again have to deal with lots of download links to the projects repository downloads. See the ownCloud Client Package download page as an living example.

Now the page had a shortcoming for apt based distributions. I became aware of it as the problem as well as the solution to it was nicely pointed out by this blog: Apt wants to have the key before it trusts the download source. OBS generates the key but its not obvious how to import.

This morning I told one of the brave OBS developers to the blog post and already now the solution is online. From now on, the download page also shows the exact command line to import the key. Isn’t that great how quickly things become better if a couple of people do a little bit work together?

Thanks a lot, guys, really cool…

Subject of the BOF will be the integration between KDE and ownCloud. As onwCloud originated from the KDE project we should meet and think together again how we can provide useful integration for the users combining the power of both technologies.

After a short overview of where the ownCloud project is going and what the recent achievements are (if participants are interested) I think we should concentrate on the technical level.

There are a couple of rough ideas already how we can integrate on a technical level, such as

• Integration of the ownCloud data storage in the KDE user experience.
• KDE features for the ownCloud Desktop Client like KWallet integration or KDE SSL certificate management.
• Synchronisation of PIM data like contacts, addresses, bookmarks or RSS feeds
• a syncing API for application data to be synced in KDE applications.
• Syncing of KDE application configurations.
• Plasma integration of ownCloud/syncing client.

I am sure there will be more we can come up with. Let me know upfront if you want me to put it on the list or even better show up at the BOF in person.

Along with that I am happy to let you know that we also released the ownCloud Desktop Client in version 1.0.2.

It is a maintenance release with a couple of important bugfixes, such as the cross platform filename encoding problem I was already talking about. Apart from that big blocker a couple of smaller, but annoying problems were fixed. Also the GUI was polished, text changes here and there and a new icon set that looks more cool and more like ownCloud.

Version 1.0.2 is also the version that is released on all big desktop platforms the first time. Now we also offer a dmg for MacOSX as well as a Windows Installer and packages for the major Linux distributions. Have fun!

If you want to talk about ownCloud, file synchronization or such, we have a booth on Linuxtag in Berlin and I will be there. I am looking forward to meeting you.

This bug took me on an interesting (well…) journey into what cross platform really means.It is about a complex thing like filenames containing special characters, my favorite example is the directory “Übergröße”. Directories and files like this could not successfully be synced with the latest client.

Windows does not utf8 by default. As far as I understood it, it does a local encoding by default. If you use normal C functions to access the filesystem, something like readdir or so, you get the names in that encoding. But, because that caused problems in some point, there is another set of functions which often start with _w (for example _wreaddir) and that has an interface to deal with wide characters. This is the way to internationalization.

So what needs to be done is to use the wide character implementations everywhere where you deal with filenames (in my case). That, in turn, requires to use wchar_t instead of normal char. But that again requires a lot of #ifdef _WIN32, if its done the naive way at every place it needs to be done. I learned that typedef helps here and a meta type is defined called _TCHAR. And also for all the needed functions, meta functions are defined which are replaced by tricky defines depending on the platform. For example _treaddir becomes readdir on linux and _wreaddir on win32, while _TCHAR becomes wchar_t on Win and char on linux. The code is written using the meta types to not poison it too much with platform ifdefs.

But that is only the first step. Wide character is not utf8! And since utf8 should be used within the software all over, a conversion from wide character to utf8 and back is needed whenever the file system is accessed. Thankfully from csyncs nature, these places are quite nicely concentrated.

All that combines into a larger patch. And now the “Übergröße” folder can be synced correctly from Windows to Linux and back. Great.

What can we learn from that? Well, easy, and once again: If you have no Qt, you’re alone. Better don’t let that happen. If you have Qt, be happy and aware of what it does for you :-)

Only a couple of weeks after the first release we could come up with a quite long changelog, larger than for a maintainance release.

First and foremost, oCC now supports SSL connections and does not store passwords any more, well, in clear text in a config file. Still there is no strong encryption for the password in (as that’s a non trivial if not impossible thing) but its not clear text any more, so accidents should not longer happen. For those who really care and put security over convenience there now is the option to not store the password at all but provide it at startup. The whole password storage will get more attention soon, I did some work on that already, also using kwallet, more on that in another post.

The whole start process of oCC, which is more complex as it might seem as libcsync expects a nice environment with a verified connection to the ownCloud, was cleaned and matured. I had to learn about QNAM but now we seem to be friends.

Efforts also went into the following areas:

• Error handling: People got confused about error messages from oCC which were not accurate here and than. I added more fine granular error reporting to libcsync so that oCC now can give even more helpful error messages.
• MacOSX: On the sprint weekend in Stuttgart, we made good progress on that platform, will also soon get a release. Domme got the most out of cmake to get dmg image building going, thanks, you rock!
• Qt 4.6: As usual one has to learn that current distributions ship not so current Qt versions. Thanks to Thomas, he put quite some effort in porting back to Qt 4.6 - so that Debians friends can also use oCC now.
• Packages: Check out the community repo on OBS: We now can provide packages for way more platforms including Ubuntu and Debian. Still your help would be appreciated in maintaining.

Speaking about the recent sprint in Stuttgart: It was great to be there, ownCloud has an open, friendly and welcoming community in which you really feel the spirit of working on something new and interesting with cool potential. I really enjoy being here,. Thanks guys for all your work and feedback :-)

Linux and Windows development should go in parallel as easy as possible.

Edit: Please also refer to the official build documentation.

Building CSync

To build mirall, csync must be built first. CSync is hosted in its upstream git repo and there is also my development branch which holds the latest changes.

Clone the csync branch into a directory. In parallel to the cloned csync dir, create a new directory buildcsync as a cmake build dir. Change into buildcsync and call cmake like this:

cmake -DCMAKE_BUILD_TYPE="Debug" ../csync

and watch its output. You probably have to fulfill some dependencies, make sure to install all the needed devel packages. You will need log4c, iniparser, sqlite3 and for the modules libssh, libsmbclient and neon for the ownCloud module. Once cmake succeeds, call make to build it. So far relaxed for Linux.

To build csync for Windows, there are a couple of possibilities. The one I chose was to cross compile with mingw under openSUSE. That way I can build for all on one devel machine under my prefered system.

For that, I installed the cross compile and mingw32 packages from the openSUSE Build Service, which really demonstrates power here. I used the mingw repository. Kudos at this point to Dominik Schmidt, a Tomahawk developer, who helped me a lot to set all up and to all people who work in OBS to maintain the mingw repo.

Basically the cross compiler and libs (eg. packages mingw32-cross-gcc, mingw32-gcc-c++ and mingw32-cross-cpp) and the dependencies for the software to build have to be installed from the mingw repo. An action item is left to dig which in detail.

After installation you should have some mingw32-tools such as mingw32-cmake which should be used to build for win.

Now create a directory win and within that again buildcsync. In there, start cmake with

mingw32-cmake -DCMAKE_BUILD_TYPE="Debug" -DWITH_LOG4C=OFF ../../csync

That should do it. I did not find log4c for Win32, so I disabled it in the cmake call. Now build it with mingw32-make and see if it creates a dll in the src subdir and csync.exe in the client dir.

Building mirall

For mirall, it works similar. Mirall uses Qt and is C++, so again a lot of packages to install. Make again sure to have the mingw32-qt packages, for example mingw32-libqt4-devel and more.

However, there are two caveats with mirall:

• the current development state of mirall needs the latest devel version of csync which we just built. I tweaked the CMakefile that way that if the mirall- and csync and build* folders are in the same directory, the csync is found by mirall cmake in the parallel dir. So I do not have to install the devel version of csync in my system.

• to build mirall for windows, it must be made sure that cmake finds the mingw32 Qt tools like moc. Since there is also the Linux moc in the system, this can confuse. Domme pointed me to a script that sets some variables correct values to prevent mixing: cat ../docmake.sh # %_mingw32_qt4_platform win32-g++-cross export QT_BINDIR=/usr/bin export BIN_PRE=i686-w64-mingw32 /usr/bin/mingw32-cmake \ -DCMAKE_BUILD_TYPE="Debug" \ -DQMAKESPEC=win32-g++-cross \ -DQT_MKSPECS_DIR:PATH=/usr/i686-w64-mingw32/sys-root/mingw/share/qt4/mkspecs \ -DQT_QT_INCLUDE_DIR=/usr/i686-w64-mingw32/sys-root/mingw/include \ -DQT_PLUGINS_DIR=/usr/i686-w64-mingw32/sys-root/mingw/lib/qt4/plugins \ -DQT_QMAKE_EXECUTABLE=${QT_BINDIR}/${BIN_PRE}-qmake \ -DQT_MOC_EXECUTABLE=${QT_BINDIR}/${BIN_PRE}-moc \ -DQT_RCC_EXECUTABLE=${QT_BINDIR}/${BIN_PRE}-rcc \ -DQT_UIC_EXECUTABLE=${QT_BINDIR}/${BIN_PRE}-uic \ -DQT_DBUSXML2CPP_EXECUTABLE=${QT_BINDIR}/qdbusxml2cpp \ -DQT_DBUSCPP2XML_EXECUTABLE=${QT_BINDIR}/qdbuscpp2xml ../../mirall

With that setup I can build both the Linux and Windows version quite easily. There is still a lot to be solved, such as automatted packaging and such. CMake as usual is a great help.

Last week I have been on CeBIT for a day, which is a huge and busy event. We had a nice preview running there with syncing between the web client and the desktop clients on Windows and Linux, as well as on an Android device. That already worked quite ok, nevertheless I got quite some bug reports since then. And I still have to work with csync upstream to get patches in there before we’re ready for a public beta.

This weekend I will be at the Chemnitzer Linuxtage together with friends from the openSUSE project. I will show some ownCloud syncing there as well, so if you like show up at the openSUSE booth for a demo and a chat :-)

That involved basically three things: A csync module, a patch in ownCloud and work on mirall.

For csync, there needs to be a module that handles the connection to ownCloud, for code see [1]. ownCloud has a WebDAV interface, and its used for this. Csync requires the the implementation of a couple of POSIX functions such as opendir, readdir, closedir, stat and these kind of things. I did that using WebDAV equivalents.

Unfortunately one method is needed for the sync algorithms which has no equivalent in WebDAV, which is utime. utime sets the modification time of a file. csync needs that to fix the time after a file was copied to the other side.  For that, I made a little patch to ownCloud to allow a PROPPATCH method on the modified time of a resource. That makes csync happy and working, but that also means that the module is not a general csync webdav module but a csync owncloud module. Named it accordingly.

The first action I started on mirall this time is a bit cleanup. First, I introduced a class  to move all the code which loads, edits and monitors the folders out of the main application class, which was long overdue. That separates the sync folder management in mirall from the GUI stuff. Furthermore I added a few helper classes such as a config file class for mirall. Ah yes, a theming class was also added to handle the future two faces character of mirall easily, as said before mirall should either appear as traditional mirall client, or as ownCloud client. Duncan and me agreed to build that from one source, the theming class supports that.

Last but not least I polished the wizard in which a user connects to his ownCloud and added a test call that immediately evaluates the url and credentials. For the curious, the code is in my work repository but I promise to move it to the ownCloud repo on gitorious soon.

So far for now, if you have suggestions for me I’d love to hear them.  Thanks :-)

[1] Klaas’ WIP branch csync